Tag Archives: automation

gitlab CI

Getting into continuous integration can be a little intimidating, here’s a good gitlab-ci.yml that I’ve adjusted over time and is pretty much drop in for my development cycle. You’ll have to add your own variables in a projects CI settings and do some SSH setup for rsync to work, I’ll cover that later.

stages:
  - test
  - build
  - user-test
  - deploy

test:
  stage: test
  before_script:
  # Find the SSH Agent
  - 'which ssh-agent'
  # Run The Agent
  - eval $(ssh-agent -s)
  # Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
  - ssh-add <(echo "$SSH_PRIVATE_KEY")

  script:
    - echo "Make Sure Commit SHA Exists, Use for revision hashing"
    - echo $CI_COMMIT_SHA
    - echo "Test Deploy To Preview"
    - rsync --update --dry-run --checksum -rlvzub --exclude ".*" --exclude "*.tgz" --exclude "Makefile.*" --backup-dir="backups/$CI_JOB_ID" -e "ssh -i ~/.ssh/id_rsa.pub" _src/ gitlab-runner@linux3:/var/www/mc-dev/$DEPLOYMENT_PREVIEW_DESTINATION/_preview
    - echo "Dry Run Deployment On Linux3"
    # - ssh gitlab-runner@linux3 -tt
    - rsync --update --dry-run --checksum -rlvzub --exclude ".*" --exclude "*.tgz*" --exclude "Makefile.*" --backup-dir="backups/$CI_JOB_ID" -e "ssh -i ~/.ssh/id_rsa.pub" _src/ gitlab-runner@linux3:/var/www/$DEPLOYMENT_DESTINATION

build:
  stage: build
  script: 
    - echo "Building The App"
    - echo $CI_COMMIT_SHA

user-test: 
  stage: user-test
  before_script:
    # Find SSH Agent
    - 'which ssh-agent'
    # Run the Agent
    - eval $(ssh-agent -s)
    #Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
    - ssh-add <(echo "$SSH_PRIVATE_KEY")
  script:
    # - echo "Update Local Preview"
    # - rsync --update --checksum -rlvzu --exclude ".*" --exclude "*.tgz" --exclude "Makefile.*" -e "ssh -i ~/.ssh/id_rsa.pub" _src/ _preview
    - echo "Create Revision Hashes"
    - sed -i "s/@@hash/$CI_COMMIT_SHA/g" _src/index.php
    - echo "Deploying to the preview folder"
    - rsync --update --checksum -rlvzub --exclude ".*" --exclude "*.tgz" --exclude "Makefile.*" --backup-dir="backups/$CI_JOB_ID" -e "ssh -i ~/.ssh/id_rsa.pub" _src/ gitlab-runner@linux3:/var/www/mc-dev/$DEPLOYMENT_PREVIEW_DESTINATION/_preview
    
  when: manual
  allow_failure: true

deploy_prod:
  stage: deploy
  before_script:
    # Find the SSH Agent
    - 'which ssh-agent'
    # Run The Agent
    - eval $(ssh-agent -s)
    # Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
    - ssh-add <(echo "$SSH_PRIVATE_KEY")

  script:
    - echo "Deploy App to Linux 3"
    # - ssh gitlab-runner@linux3 -tt
    - echo "Create Revision Hashes"
    - sed -i "s/@@hash/$CI_COMMIT_SHA/g" _src/index.php
    - echo "Copying Files"
    - rsync --update --checksum -rlvzub --exclude ".*" --exclude "*.tgz*" --exclude "Makefile.*" --backup-dir="backups/$CI_JOB_ID" -e "ssh -i ~/.ssh/id_rsa.pub" _src/ gitlab-runner@linux3:/var/www/$DEPLOYMENT_DESTINATION

  environment:
    name: production
    url: http://linux3/$DEPLOYMENT_DESTINATION
  only:
    - master
    - merge-requests
  # Uncomment below to turn on manual deployment
  # when: manual
  allow_failure: false

 

Keep an eye out for more coverage of setting up rsync over ssh.

Markdown -> Email

I love markdown, I like the simplicity of it, the consistent formatting and all the edge features that are possible. So sifting through my snippet collection I found this beaut in my GitHub and my Gitlab snippet collections so it must be good right?


#!/usr/bin/env python

'''
Send an multipart email with HTML and plain text alternatives. The message
should be constructed as a plain-text file of the following format:

From: Your Name 
To: Recipient One 
Subject: Your subject line
---
Markdown content here
The script accepts content from stdin and, by default, prints the raw
generated email content to stdout.
Preview your message on OS X by invoking the script with `-p` or
`--preview`, and it will open in your default mail client.
To send the message, invoke with `-s` or `--send`. You must have a
JSON file in your home directory named `.markdown-to-email.json`
with the following keys:
{
"username": "smtp-username",
"smtp": "smtp.gmail.com:587",
"password": "your-password"
}
Enjoy!
'''
import os
import sys
import json
import argparse
import smtplib
import subprocess

from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

try:
import pygments
import markdown
except ImportError:
print 'This script requires pygements and markdown to be installed.'
print 'Please:'
print ' pip install pygments markdown'
sys.exit(0)

# define arguments
parser = argparse.ArgumentParser(description='Format and send markdown-based emails.',
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=__doc__)
parser.add_argument('-p', '--preview', action='store_true',
help='Preview the email in Apple Mail.')
parser.add_argument('-s', '--send', action='store_true',
help='Send the email using your configuration.')
args = parser.parse_args()

# read in raw message content
raw_content = sys.stdin.read()

# split out the headers from the markdown message body
header_content, markdown_content = raw_content.split('--', 1)

# render the markdown into HTML
css = subprocess.check_output(['pygmentize', '-S', 'default', '-f', 'html'])
markdown_content = markdown_content.strip()
html_content = markdown.markdown(markdown_content, ['extra', 'codehilite'])
html_content = ''+css+''+html_content

# create a multipart email message
message = MIMEMultipart('alternative')

# parse the headers
headers = {}
for line in header_content.strip().split('\n'):
if not line.strip(): continue
key, value = line.split(':', 1)
headers[key.strip()] = value.strip()

# set the headers
message['To'] = headers.get('To', '')
message['From'] = headers.get('From', '')
message['Subject'] = headers.get('Subject', 'No subject')

# attach the message parts
message.attach(MIMEText(markdown_content, 'plain'))
message.attach(MIMEText(html_content, 'html'))

if args.send:
to = message['To'].split(', ')

with open(os.path.expanduser('~/.markdown-to-email.json'), 'rb') as f:
config = json.loads(f.read())
server = smtplib.SMTP(config['smtp'])
server.starttls()
server.login(config['username'], config['password'])
server.sendmail(message['From'], to, message.as_string())
server.quit()
elif args.preview:
open('/tmp/preview.eml', 'w').write(message.as_string())
os.system('open -a Mail /tmp/preview.eml')
else:
print message.as_string()

 

It has some external dependencies but creating rich html emails from simple markdown base is pretty sweet in my book.