Using scp to copy from a server and to a server

Copy files from the server to your pc

You need ssh access, view this post to do that:

Add ssh keys and ssh access to a linux server

The command to scp from server to your computer

1
sudo scp -i ~/.ssh/id_rsa user@server:A B

eg

1
sudo scp -i ~/.ssh/id_rsa user@server:/the/filepath/on/server/**/* /filepath/of/computer

/**/* refers to all files and subfolders

Copy files from your pc to the server

The normal way to scp, where you copy files from your computer to the server

1
scp -r A user@server:B

eg:

1
scp -r /path/on/your/computer/ user@server:/path/to/server/destination

-r means recursive (so all files and sub folders)

Wildcards

You can use wildcards to match with a file extension

1
eg *.php

That will copy all php files only.

Create a temp directory with ansible on your local computer

I have a linux environment set up, and I have ansible installed. To see how to setup ansible from scratch follow this blog entry:

Setting up ubuntu linux for ansible development

Some tips that helped me while I was researching ansible

(I’m still learning so this may or may not be right, use it more as stepping stones to progress your own knowledge)

Yml

Ansible makes use of .yml files. Pronounced “yeah-mil”. These files are indent sensitive and you seperate commands be new lines and indents. If you are unfamiliar with yml structures I suggest googling a bit for more information on yml files are structured.

Hosts

There is a global ansible hosts file.

/etc/ansible/hosts

However I always recommend against editing global files. Setup a local inventory file rather.

In your hosts files you declare host level variables that indicate the addresses/locations you would like to access and alter from.

Playbooks

A playbook defines individual plays to be executed in a predefined order

Play

A play is a single task or action to be execute

Modules

A play can use modules to achieve tasks or actions

Lets get started with a simple ansible script to create a directory

Go to a new directory where you would like to create the ansible script and create a folder called “practice”.

In this folder create a yml file called “test.yml”.

Note you can name the yml file anything you like, I’ve just chosen test for this example.

Then create an inventory folder for you hosts file. In the inventory folder add a “hosts” file (no extension).

1
2
3
4
practice/
    inventory/
        hosts
    test.yml

Add hosts

In your “hosts” file add the following:

1
2
[local]
localhost ansible_connection=local

This indicates a variable “local” with the value localhost ansible_connection=local. This tells ansible that you are working on your local machine.

Point to local in test.yml

Add these lines to your test.yml file in order to run your first ansible command.

1
2
- hosts:
  - local

You can check that ansible picks up this configuration by running:

1
<p class="p1">ansible-playbook -i inventory/hosts test.yml --list-hosts</p>

And you should get the response:

1
2
hosts (1):
    localhost

The -i refers to:

-i INVENTORY, –inventory-file=INVENTORY
specify inventory host path
(default=/etc/ansible/hosts) or comma separated host
list.

Then the inventory/hosts indicates the “hosts” file you created in the inventory directory.

The all indicates that all the listed hosts will be displayed. You could also say:

1
ansible-playbook -i inventory/hosts --limit localhost test.yml --list-hosts

And that would filter to just the “local” section. In this case it would return the same result:

1
2
hosts (1):
    localhost

The playbook

Lets start adding a task. Open your test.yml file in you practice folder, and add the following lines:

1
2
[local]
localhost ansible_connection=local

This indicates the environment you wish to alter. Run the playbook:

1
ansible-playbook test.yml -i inventory/hosts

If you get this message:

skipping: no hosts matched

Just make sure you indicated the “-i inventory/hosts” part.

Add tasks

Now we will begin adding the task of creating a new folder. We will be making use of an ansible module called “file”.

Add the following code to your test.yml file:

1
2
3
4
   
  tasks:
    - name: create a dir called temp
      file: path=temp state=directory mode=0755

“name” is referring to the name of the task, this will display in the response on the command line

“file” is the module you are using, and then path, state and mode are configuration values you set for the task

You should now have in your test.yml file:

1
2
3
4
5
- hosts:
  - local
  tasks:
    - name: create a dir called temp
      file: path=temp state=directory mode=0755

Run your ansible playbook

1
ansible-playbook test.yml -i inventory/hosts

You should now see a temp folder inside your practice folder you created.

And that’s it. As simple as that.

Experiment with more file modules

http://docs.ansible.com/ansible/list_of_files_modules.html

Npm node-http2 error ENOENT

I started up an existing full stack project on a new ubuntu installation. I ran npm install and bower install and on grunt serve I kept getting a similar error to this:

npm ERR! enoent ENOENT: no such file or directory

The error seemed to be triggered by node-http2 specifically.

I tried several approaches, I upgraded node, upgraded grunt. I uninstalled and reinstalled. Cleared cache.

What I finally realised the problem was, I needed to install yo. The project had been created using the yeoman generator.

Checklist:

  • Make sure npm is installed
  • Make sure bower is installed
  • Make sure grunt is installed
  • Make sure yo is installed

These are some additional commands I ran

Clear npm cache and reinstall npm

sudo npm cache clean -f

sudo npm install -g n

sudo n stable

 

Deleted the \tmp folder and receiving “bash: cannot create temp file for here-document: Permission denied” Error

I was trying to delete ~/tmp folder and accidentally deleted the /tmp folder. Then when I tried to ls a folder structure I kept getting this error:

bash: cannot create temp file for here-document: Permission denied

To fix the error I ran the following commands:

1
2
3
4
5
6
7
cd /

mkdir tmp

sudo chown root:root /tmp

sudo chmod 1777 /tmp

 

Linux command to restart network manager

I have been using Ubuntu a lot more, and I often need to switch between different network connections and go on and off of a network proxy. All that jumping around tends to affect my connectivity to the different networks. I find that the connecting part tends to hang and I cannot connect to a network anymore.

I found this handy command to help with that:

sudo service network-manager restart

After the network manager restarted on Ubuntu I was able to connect to a network with no delay.

Tips to get docker set up on Ubuntu

So I recently tried setting up docker on ubuntu and ran into a few hurdles, below is the checklist of what needed to be setup or installed on Ubuntu in order to run a docker project.

Make sure these are installed/Setup

docker
docker-compose
virtualbox
docker-machine

Setting up the docker-machine default ip

Then using docker machine you need to setup an IP to run in a web browser.

There is more thorough documentation on the docker website. The summary of what I found worked:

Run the following commands:

docker-machine create –driver virtualbox default
docker-machine env default
eval $(docker-machine env default)

Get the ip address of default

docker-machine ip default

That ip is what you will be using in your browser to view the docker files.

When you run your docker container:

docker-compose up

Then you will be provided with a port number. Eg 127.0.0.1:8000

Then you use the ip above and that port number.

See the other docker posts for solutions to some of the errors I ran into and fixed.

Configuring github to work through a proxy

Edit the .gitconfig file on Windows to work with a proxy

Open your .gitconfig file. You can find this file in your C:/Users/[your user] folder.

Edit the .gitconfig file on Ubuntu/Linux to work with a proxy

Open your .gitconfig file. You can find this file in your home ~/ folder.

You can type in the command line:

sudo nano ~/.gitconfig

Edit the .gitconfig file

Insert the following lines:

[http]
proxy = http://username:password@proxyurl:port
[https]
proxy = http://username:password@proxyurl:port
sslVerify = false

Username would be your proxy username.

Password would be your proxy password.

Proxyurl would be the proxy url.

Port would be the proxy port.

View your git settings

You can check your git settings in the command line by running the following:

git config -l

 

Configuring a proxy on windows and linux

I have a daily battle with a proxy. Each day I think, “today this proxy will be tamed”. Only to have the proxy stop working and I have to start figuring it out from scratch.

Below is what I go through each time I experience problems with proxy connection.

Connecting to a proxy via command line:

Windows

Configure your windows environment to work through a proxy

Environment variables

One way is to update your environment variables.

Open a command console, and in it add these two settings:

set http_proxy=”http://proxyUsername:proxyPassword@proxyUrl:port”
set https_proxy=”’http://proxyUsername:proxyPassword@proxyUrl:port”

View environment variables

To view your system settings enter in your command console:

set

Using “set” is more like a “cache” setting, it will go away when you close the command console, or restart your computer. To set it permanently, either manually add it to your environment variables. Or use setx.

setx /M http_proxy “http://proxyUsername:proxyPassword@proxyUrl:port”
setx /M https_proxy “http://proxyUsername:proxyPassword@proxyUrl:port”

A note on the value ”http://proxyUsername:proxyPassword@proxyUrl:port”

This format can be:

  • ”http://proxyUsername:proxyPassword@proxyUrl:port”
  • ”http://proxyUrl:port”

Another note: ping will not work with a proxy. (Well for now I haven’t found a way for it to work yet.) From the research I have done to date, ping works at a different level to the proxy. So just a head’s up, if you use ping to test if your proxy settings now allow internet access, you will be going in circles.

What should work is if you use a tool like vagrant for example and you need to download a box.

If this doesn’t work, and you use a tool that needs internet acces through the command line and it still cannot connect to the internet. Make double-triple sure your proxy details are 100% correct. Make sure you are connected to the internet.

And try the below:

LAN Settings

Setting the proxy on the lan settings. I have found two ways of setting this, one way is through a web browser like Chrome. And the other is through the control panel.

Using chrome

Go to Settings.
Show advanced settings.
In the Network section click “Change proxy settings”.
In the window that opens up click on “LAN settings”.
Select “Use a proxy server for your LAN…”
Add the proxy address, add the proxy port.

Additional notes on using windows for proxy

  • Often something doesn’t work because you need to open a new console window for to take effect (eg changing environment variables).
  • Often something doesn’t work because you nee the entire system to be restarted (this is always my last go to, if all else fails, restart).

Proxy is still not working on windows

If you have done all of the above and you still do not have internet acces through the proxy, you need to check if your proxy itself is not blocking the access.

Setting up proxy details on Linux

This is still a work in progress, below is what I’ve figured out so far:
To set the system variables on linux:

export http_proxy=”http://proxyUsername:proxyPassword@proxyUrl:port”
export https_proxy=”http://proxyUsername:proxyPassword@proxyUrl:port”