My blog moved to giltalb.io !
https://demeringo.gitlab.io/blog/

• Overiew (linux.com)
• gdd tutorial
• ddrescue
• tools review (arch wiki)

English

wpscan

wpscan on github

The quickest way to use it seems to be docker.

docker pull wpscanteam/wpscan

# minimal run
docker run --rm wpscanteam/wpscan -u https://le-moulin-de-verre.com/fieldnotes/
# more details options (enumerate vulnerabilities of identify plugins)
docker run --rm wpscanteam/wpscan -u https://le-moulin-de-verre.com/fieldnotes/ --follow-redirection --update --enumerate

All start with a question: can we run Windows software in a Docker container ?
Yes and No… unsurprisingly it depends on the host that executes the container.

A article from Azure CTO that explains MS Azure strategy around Docker: https://azure.microsoft.com/en-us/blog/containers-docker-windows-and-trends/

Working in a corporate environment, it may come that web access is filtered through a proxy… and even an NTLM one which requiere MS-style authentication.

If you also are lucky enough to run Docker inside a Linux Virtual machine (not natively on Windows), your docker builds or package update operations may fail at first.

You may ease the building and running of docker containers by using an intermediate proxy (cntlm) that will authenticate your and forward your requests against the coprorate proxy, while acting as simpler (unanthenticated) proxy for your doacker daemon or containers.

Build images behind a proxy

My environment:

- Coporate NTLM proxy
- Windows 7 (running cntlm proxy properly configured, listenning to 127.0.0.1:9128, and NOT in gateway mode)
- Virtual Box 
- Centos 7 
- Docker 

Make docker daemon use the proxy of the Windows host

Not hard but a bit complex:

The first thing to take care of with all these virtualization layers is that 127.0.0.1 means something different for each layer !

The tricky part is to understand that Virtual box creates a specific NIC inside CentOS (address is 10.0.2.x with a default gateway of 10.0.2.2). The 10.0.2.2 IP address is redirected by Virtual Box to the 127.0.0.1 of Windows, on which cntlm proxy is listenning.

Last thing, the daemon needs to be aware of proxy at build time (download an image from web), while apt-get or yum may need to be aware of proxy at build time but also possibly at runtime.

In short, from docker daemon perspective, the proxy is 10.0.2.2:9128

Docker daemon setup

Create

/etc/systemd/system/docker.service.d/docker_http_proxy.conf

With content:

[Service]
Environment="HTTP_PROXY=http://10.0.2.2:9128/"
Environment="HTTPS_PROXY=https://10.0.2.2:9128/"

Then reload configuration and restart daemon

sudo systemctl daemon-reload
sudo systemctl restart docker

Build the image

Then you should run the build using buildtime-only proxy configuration:

docker build --build-arg http_proxy=http://10.0.2.2:9128 --build-arg https_proxy=https://10.0.2.2:9128 -t my_app .

TBC: is --build-args needed for yum update only ?

Run containers behind a proxy

Here the context is a bit different: we want a running container to access the web. Generraly speaking, it may not be the best idea in production environments, but it may still greatly help you to test or develop inside container.

Quick and dirty: rely on another container to provide the proxy

The simplest way is to use an already existing container, made specifically to provide transparent cntlm proxy service to others through Docker -link capability. It allows your application to address the proxy by name.

You should rather build this container from source (rather than blindly downloading it) as you will entrust it sufficiently with your logins and passwords.

Several cntml proxies are available, I choose to use jaschac/cntlm.

Download / build image with:

docker pull jaschac/cntlm

Start jashac/cntlm container:

docker run --name cntlm -e CNTLM_USERNAME=your_username -e CNTLM_DOMAIN=your_domain -e CNTLM_PROXY_URL=your_parent_ proxy -e CNTLM_PROXY_PORT=your_parent_proxy_port -e CNTLM_PASSNTLMv2=yourhash -d jaschac/cntlm

Then start your applciation container with:

docker run --link cntlm:cntlm -it my_app /bin/bash

Note the –link option that allow your application to resolve the cntlm host destite in another container.

Optional: set the proxy env in the dockerfile:

ENV http_proxy http://cntlm:3128
ENV https_proxy https://cntlm:3128

Make your container aware of the external proxy

Use the same address (corresponding to Windows 127.0.0.1 through Virtual box) to let your container access the Windows cntlm proxy at runtime.

TBD: remains to be tested

ENV http_proxy http://10.0.2.2:9128
ENV https_proxy https://10.0.2.2:9128

Albert uses Sharepoint 2013 to store several GB of documents.

Albert needs to be able to list and possibly extract the documents in a comuter readable format, like ls would do on Unix. Of Course, he is a bit in a rush and has nothing but a corporate laptop without much privileges to do the job….

To make things simpler, although the Sharepoint site is hosted by his company, he has only a standard user web access and no specific provilege on the server or a powershell environment that would allow to interrogate it.

What’s next ?

Using Python, we can quite simply rely on Sharepoint REST API to perform basic queries.

Setup

Use winpyton

A portable Pyhton

Installing Python 3 and needed libraries may be tricky in a corporate environment, in particular when the web is accessed through a NTLM proxy that also rewrites SSL certificates.

I ended up installing Winpython (portable environment that bundles a recent PIP version).

using pip to access pypi.org behind a proxy

By default something like pip --proxy=http://user:pass@proxyAddress:proxyPort
should do the trick.
I rely on a local CNTML proxy to authenticate my requests to the corporate proxy, as this avoid having to manage the account/password  in the commands
pip --proxy=http://localhost:3128 install my_special_package

However in my case the proxy rewrites SSL certificates and sign them with a corporate cert. I need to tell pip to ignore pypi.org certificate validation.

C:\WinPython-64bit-3.4.4.2\scripts>pip --proxy=http://localhost:3128 --trusted-host pypi.python.org install my_special_package

Ideally we should be able to tell pip to trust the corporate certifcate (did not work in my case though, possibly due to a problem of cascading certs.):

pip --cert /etc/ssl/certs/FOO_Root_CA.pem install my_special_package

Installing dependencies

We will also need to authicate through NTLM to access the Sharepoint server(basic auth as shown in most online sample does not work here, we get error 401).

pip --proxy=http://localhost:3128 --trusted-host pypi.python.org install requests
# requests_ntlm package to authenticate http requests
pip --proxy=http://localhost:3128 --trusted-host pypi.python.org install requests_ntlm
# sharepoint lib... that did not work though
pip --proxy=http://localhost:3128 --trusted-host pypi.python.org install sharepoint

the script

import requests
import json
from requests.auth import HTTPBasicAuth
from requests_ntlm import HttpNtlmAuth
import getpass

# 24-06-2016 - olivier de Meringo... and a lot of nice guys on the web.
#
# Experimental listing files and directories and some metadata from a sharepoint 2013 site.
# More or less like ls-al on sharepoint where you only have non-admin access (i.e. only REST API).


# ITEMS = "http://mysite.corp/site/subsitename/_api/lists/getbytitle('ESD repository')/items"
# SITE = "http://http://mysite.corp/site/subsitename/_api/web/getfilebyserverrelativeurl('/')"
#USER_FOLDERS= "http://mysite.corp/site/subsitename/_api/web/getfolderbyserverrelativeurl('/site/subsitename/Documents')/Folders"

MYDOC_REPOSITORY = "http://mysite.corp/site/subsitename/_api/web/getfolderbyserverrelativeurl('/site/subsitename/Documents')"


# User name with nt domain in case of NTLM (USERNAME = "mydomain\\mylogin")
USERNAME = "mydomain\\mylogin"
# prompt password
PASSWORD = getpass.getpass('Password for '+USERNAME+':')




def get_json_ntlm(query):
headers = {'accept': 'application/json;odata=verbose'}
r = requests.get(query, auth=HttpNtlmAuth(USERNAME,PASSWORD), headers=headers)
return r

# display json in human readable format
def pp_json(json_thing, sort=True, indents=4):
if type(json_thing) is str:
print(json.dumps(json.loads(json_thing), sort_keys=sort, indent=indents))
else:
print(json.dumps(json_thing, sort_keys=sort, indent=indents))
return None



def print_files_recursivelly(folderURI):
# get current folder as json
folderJson = get_json_ntlm(folderURI).json()

# gets the URI that describes the files of this folder
filesURI = folderJson["d"]["Files"]["__deferred"]["uri"]
print_files_detail(filesURI)

# Continue on subfolders
subURI = folderJson["d"]["Folders"]["__deferred"]["uri"]
subfolders = get_json_ntlm(subURI).json()
for sub in subfolders["d"]["results"]:
print_files_recursivelly (sub["__metadata"]["uri"])
return None

def print_files_detail(filesURI):
files = get_json_ntlm(filesURI).json()
for afile in files["d"]["results"]:
print(afile["Name"]+ ";"+ afile["ServerRelativeUrl"]+";"+ afile["Length"] +";"+ afile["TimeCreated"] +";"+ afile["TimeLastModified"] +";")
return None

def print_folder_detail(folderJson):
# print it's name and relative URL
print ( "{0} ({1} child(s))".format(folderJson["d"]["ServerRelativeUrl"], folderJson["d"]["ItemCount"]))
return None

def print_folder_hierarchy(folderURI):
# get the folder as json
folderJson = get_json_ntlm(folderURI).json()
print_folder_detail(folderJson)

# gets children files
# filesURI = folderJsonRes["d"]["Files"]["__deferred"]["uri"]
# print_files_detail(filesURI)

# get subfolders uri
subURI = folderJson["d"]["Folders"]["__deferred"]["uri"]
subfolders = get_json_ntlm(subURI).json()
for sub in subfolders["d"]["results"]:
print_folder_hierarchy (sub["__metadata"]["uri"])
return None

print_folder_hierarchy(MYDOC_REPOSITORY)
print("-------------------------------------------------")
print_files_recursivelly(MYDOC_REPOSITORY)

quick notes – improvements

I should rather use python-sharepoint library

Python-sharepoint library is expected abstract the REST API but I did not manage to use it, possibly due to a problem with NTLM opener.

See [https://github.com/ox-it/python-sharepoint/issues/3]

Use CAML to filter queries

REST API can use CAML to reduce and optimize queries. However, it will requiere that REST queries are send through POST (instead of GET) which may conplexify the code…. but definitly somehing to explore.

See
– [http://sharepoint.stackexchange.com/questions/78612/why-do-i-have-to-use-post-for-rest-queries]
– [https://chuvash.eu/2014/03/25/using-caml-with-sharepoint-rest-api/]

Json format reference for files and folders

• Files collections: [https://msdn.microsoft.com/en-us/library/office/dn450841.aspx#bk_FileCollection]
• Folder collections: [https://msdn.microsoft.com/en-us/library/office/dn450841.aspx#bk_FolderCollection]
• Folders properties: [https://msdn.microsoft.com/en-us/library/office/dn450841.aspx#bk_FolderProperties]

usefull queries

Get All Lists “`http://server/site/_api/lists“

Get All List Items From a Single List`:  http://server/site/_api/lists/getbytitle(‘listname’)/items

Get a Single List Item http://server/site/_api/lists/getbytitle(‘listname’)/items
Get Back Certain Columns http://server/site/_api/lists/getbytitle(‘listname’)/items?$select=Title,Id
Order Your Results  ““http://server/site/_api/lists/getbytitle(‘listname’)/items?$orderby=Title“`

The issue: CNTLM only works if started manually

Daemon does not executes properly (but service start command returns no error).
[root@server ~]# service cntlmd start

However cntml works fine when started from terminal with same configuration.
[root@server ~]# cntlm -c /etc/cntlm.conf

To debug, the service logs are in /var/log/messages

A first attempt…

In my case, the user executing the service is not root but cntlm and it does not exists or cannot create the PID file.

• open /etc/sysconfig/cntlmd, look for RUNAS=cntlm, and also the location of PID file.

• create the cntml user and the PID directory with correct rights.

[root@server ~]# adduser cntlm
[root@server ~]# mkdir /var/run/cntlm
[root@server ~]# chown cntlm /var/run/cntlm/

Restart the service… and all is fine… until i restart the server. After reboot I notice that the freshly created PID directtory /var/run/cntlm has disappeared.

The reason is that /var/run is of type tmpfs and not persisted accross reboots !

A persistent solution

Enters tmpfiles.d…

We can use systemd-tmpfiles (manual) to automate the creation or cleaning of directories at boot time.

See  /usr/lib/tmpfiles.d/ for example of existing configs (and write to /etc/tmpfiles.d/* override them).

Create an config for cntlm:

[root@server ~]# gvim /etc/tmpfiles.d/cntlmd.conf

#Type Path Mode UID  GID  Age Argument
d    /var/run/cntlm   0755 cntlm cntlm - -

Test it manually before reboot.

systemd-tmpfiles --create /etc/tmpfiles.d/cntlmd.conf

References

The blog  that pointed me to the right direction. Read the comments that offer the best solution:  [https://blog.hqcodeshop.fi/archives/93-Handling-varrun-with-systemd.html]

[https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html]

http://www.wijngaard.org/hardening-access-to-your-synology-diskstation-and-prepare/

http://bpmsg.com/how-to-make-your-synology-disk-station-nas-more-secure/

Quick list of security – related links:

General list:
http://www.troyhunt.com/2015/09/troys-ultimate-list-of-security-links.html?m=1

Another one:
https://github.com/enaqx/awesome-pentest/blob/master/README.md

Open security training:
http://opensecuritytraining.info/

Certifications and trainings from Offensive Security (including free Metasploit training)
https://www.offensive-security.com/

Test your email account for known breach.
https://haveibeenpwned.com/

Summary

Latest Eclipse version (LUNA, MARS) are not always able to access internet through an NTLM proxy.

This makes updating Eclipse or installing a new feature pretty impossible.

When the web proxy requires NTLM authentication, Eclipse fails to connect and returns a authentication
error message even with various configurations:
– using native proxy support fails
– using manual settings (and providing my Windows domain, username, password) fails.

Typical error:

Some sites could not be found.  See the error log for more detail.
HTTP Proxy Authentication Required:<a href="http://download.eclipse.org/releases/mars/content.xml">http://download.eclipse.org/releases/mars/content.xml</a>
Proxy Authentication Required
HTTP Proxy Authentication Required:<a href="http://download.eclipse.org/eclipse/updates/4.5/content.xml">http://download.eclipse.org/eclipse/updates/4.5/content.xml</a>
Proxy Authentication Required
HTTP Proxy Authentication Required:<a href="http://download.eclipse.org/webtools/repository/mars/content.xml">http://download.eclipse.org/webtools/repository/mars/content.xml</a>
Proxy Authentication Required
HTTP Proxy Authentication Required:<a href="http://download.eclipse.org/mylyn/releases/mars/content.xml">http://download.eclipse.org/mylyn/releases/mars/content.xml</a>
Proxy Authentication Required

In preceding versions, I was able to make Eclipse work by using options in the ‘Eclipse.ini’ file but this do not work anymore.

Another workaround consist to replace the version of HTTPClient library of eclipse (either direcly by changing a jar or by installing – offline- a custom plugin that does it for us. I would not recommend this solution. It helps first later conflicts with updates or additional package.

Bug references here:
– Bug ID + link

Using CNTML as a local proxy

CNTML ([http://cntlm.sourceforge.net/])is a small local proxy, that is able to authenticate you against the Enterprise NTLM proxy once for all and forward your requests without additional authentication.

It appears that Ecplise is ablte to traverse an NTLM proxy but not to authenticate against it.

I deploy CNTML on a local machine, provide it with my NTLM crendentials and the target enterprise proxy.
– CNTML is started and listen to localhost:3128
– Eclipse is configured to send traffic to localhost:3128
– CNTML authenticate against corporateProxy:3128
– CNTML forwards requests to corporateProxy:3128
– … Eclipse gets the response.

CNTML setup

Plenty of detailled examples through the web, see [http://stormpoopersmith.com/2012/03/20/using-applications-behind-a-corporate-proxy/] for windows.

A summary:

• DL/Install CNTML
• Update cntlm.ini (or /etc/cntml.conf)
    * Add target proxy + port
    * Restrict to localhost
    * update username, domain , do not provide clear password there (exept for test)

• run cntlm -c cntlm.ini -I -M <a href="http://google.com/">http://google.com</a> to be prompted for password

• cntml return hashed password, store it to cntml.ini

sample CNTLM sartup bat

Use the following bat file if cannot setup a service.

set cntlmexe=C:\apps\cntlm-0.92.3\cntlm.exe
set cntlmconf=C:\apps\cntlm-0.92.3\cntlm.ini

REM ------------- start normally and verbose
%cntlmexe% -v -c %cntlmconf%

REM ------------- start with password prompt to produce a hash
REM %cntlmexe% -v -I -H -c %cntlmconf%

REM ------------- start with password prompt to test
REM %cntlmexe% -I -c %cntlmconf% -M<a href="http://www.google.com/">http://www.google.com</a>
pause

Configure Eclipse to use local proxy

Set up Eclipse to use localhost:3128 as a proxy.