Content

OSINT
open-source intelligence (OSINT - wikipedia)

The Pyramid of Pain
Knowlesys - OSINT realization - looks like resource which describes osint in general

Internet is based on:

  1. Hierarchy of DNS names (tree hierarchy)
  2. RIPE databases - exists 5 regions (Europe, Central Asis; North America; Asia, Pacific; Latin America, Caribbean; Africa) each region has its own ip-address pools and each region gives sub-pools to other instances (company or provider or country or …)
  3. Set of autonomous systems - AS. (these has no hierarchy)
  4. SSL certificate chains

Metadata concept

  • by what? the file was created/changed - software type (e.g. MSWord, ImageMagick, …)
  • by whom? the file was created/changed - username, impersonalization
  • computer name, where file was created/changed
  • when? the file was created/changed - date/time
  • where? the file was located - path disclosure
  • e-mail addresses
  • ip-addresses
  • dns-names and subdomains

Most popular assests searched for compromisation:

  • an unpatched server connected to the Internet
  • an individual


Tools

Awesomeness

  • OSINT Framework - awesome collection of various tools for OSINT (Open Source Intelligence)




  • publicwww - find any alphanumeric snippet, signature or keyword in the web pages HTML, JS and CSS code
  • nerdydata.com - quality leads from all over the web

OSINT multifunctional tools / frameworks

  • only subdomain enum:

    • Sublist3r - fast subdomains enumeration tool for penetration testers - aggregates output from lots of sources (google, crt.sh, bing, virustotal, …)
      python sublist3r.py -d example.com - passive
      python sublist3r.py -b -v -d example.com - active
    • subfinder (passive) - a subdomain discovery tool that discovers valid subdomains for websites
      better use docker
    • censys-subdomain-finder (passive) - enumeration using the crt logs (crt.sh)
      python censys_subdomain_finder.py --censys-api-id [API_ID] --censys-api-secret [API_SECRET] example.com
      censys-enumeration (passive) - a script to extract subdomains/emails for a given domain using SSL/TLS certificates dataset on Censys (json output)
      python censys_enumiration.py --verbose --subdomains --emails domains.txt
    • amass (passive with dns or active) - in-depth subdomain enumeration
      purely passive: ... -nodns ...
      passive: amass -v -ip -min-for-recursive 3 -log ~/amass.log -d example.com,
      has active methods: -active -brute
    • knockpy (active) - subdomain scan
      knockpy example.com
    • enumall.py (passive + bruteforce) - automation of recon-ng subdomain discovery
      ./enumall.py example.com
      ./enumall.py -a example.com

    Not all available technics are used by these tools, e.g. you can check specific technics from subdomain enumerate category (e.g. CSP analysis for subdomain search)

  • full-featured tools:

    • aquatone - a tool for domain flyovers
      Add keys: aquatone-discover --set-key [censys_id, censys_secret, shodan, passivetotal_key, passivetotal_secret, virustotal, riddler_username, riddler_password] [VALUE]
      aquatone-discover --domain example.com --threads 25 - subdomain enumeration
      aquatone-scan --domain example.com --ports large - enumeration common ports, used for web-services
      aquatone-gather --domain example.com - retrieve and save HTTP response headers and make screenshots
      aquatone-takeover --domain example.com - check subdomain-takeover situations

    • datasploit (passive + active) - osint + active scans = HTML report
      datasploit -d example.com

    • fast analysis

      • domain_analyzer - search all info about domain
      • domain-profiler - a tool that uses information from whois, DNS, SSL, ASN, …
      • lazyrecon (active) - sublist3r and certspotter + screenshots + grab response header + nmap + dirsearch = generate HTML report
  • theHarvester (passive + active) - e-mail, subdomain and people names harvester
    python theHarvester.py -b all -d example.com
  • DMitry (active + port scan) - gather as much information as possible about a host. Base functionality is able to gather possible subdomains, email addresses, uptime information, tcp port scan, whois lookups, …
    dmitry -i -w -n -s -e example.com
    with port scan: dmitry -i -w -n -s -e -p -b -t 2 example.com


  • web-spidering:

    • BlackWidow - web-spider
      /usr/share/BlackWidow/blackwidow -d example.com -l 5
    • Photon - light web-spider
      photon.py -u http://example.com -l 5 -d 0 -t 10
    • blacksheepwall (based on CommonCrawl - grep the internet)
      blacksheepwall -cmn-crawl CC-MAIN-2018-13-index -domain sberbank.ru




  • intrigue.io - official site, github
  • spiderfoot – open source intelligence automation tool for process of gathering intelligence about a given target, which may be an IP address, domain name, hostname or network subnet

  • recon-ng (kali linux) - good (and huge) tool for various reconnaissance vectors

    usage sample 
    workspaces; workspaces add example.com
show modules

# certificate transparency query:
use recon/domains-hosts/certificate_transparency; show info; set SOURCE example.com;
run

# netcraft
use netcraft; set source example.com;
run

# Resolve Hosts, get IPs, GEO and report:
show hosts
use resolve; use recon/hosts-hosts/resolve; run
use freegeoip; run
use report; use reporting/xlsx; run


scanning tools

  • subresolve - resolve and quickly portscan a list of sub-domains

Subdomain / ip / e-mail harvesting / enumirate / etc. (concrete tools)

Subdomain enumiration
process of exposing subdomains of one or more domains


network recon

subdomain recon

Categorial/concrete tools/attacks:

  • CloudFail - utilize misconfigured DNS and old database records to find hidden IP’s behind the CloudFlare network
    python cloudfail.py --target rublacklist.net

subdomain enumerate


  • domains-from-csp - a script to extract domain names from Content Security Policy(CSP) headers
    python csp_parser.py -r http://example.com


Everything beneath can be done faster if you will use frameworks and other complex tools

  • Subject Alternative Name (SAN) - X509 extension to provide different names of the subject (subdomains) in one certificate

    Even if there is non-resolvable subdomain, probably admins use the same certificate for intranet connections.

  • Forward DNS

  • zone transfer - does DNS server expose a full DNS zone? (via AFXR) (AXFR zone transfer scan (by sergeybelove))

    dig axfr zonetransfer.me @nsztm1.digi.ninja
host -t axfr zonetransfer.me nsztm1.digi.ninja
host -avl zonetransfer.me nsztm1.digi.ninja
nslookup -query=AXFR zonetransfer.me nsztm1.digi.ninja
    • fierce -dns zonetransfer.me
    • dnsrecon -a -d zonetransfer.me

  • NSEC walking attack - enumerates DNSSEC-signed zones
    Take your DNSSEC with a grain of salt

    • apt-get install ldnsutils

      • ldns-stroll @ns1.insecuredns.com insecuredns.com
      • ldns-walk @ns1.insecuredns.com insecuredns.com
    • nsec3map – DNSSEC Zone Enumerator – позволяет перебрать содержимое всей доменной зоны и найти поддоменты, если на dns сервере работает dnssec (https://github.com/anonion0/nsec3map)
    • nsec3walker
    • nmap -sSU -p 53 --script dns-nsec-enum --script-args dns-nsec-enum.domains=example.com <target>
      nmap -sSU -p 53 --script dns-nsec3-enum --script-args dns-nsec-enum.domains=example.com <target>

subdomain bruteforce

Comparison of subdomain bruteforce tools: massdns, gobuster, dns-paraller-prober, blacksheepwall, subbrute (pic)
SecLists - check bruteforce lists
compiled GIANT subdomain wordlist (march 2018)

  • massdns
  • fierce
    fierce -dns zonetransfer.me
    fierce -dns zonetransfer.me -wordlist /path/to/wordlist.txt - for custom wordlist
  • subbrute
  • dnsrecon
    dnsrecon -d example.com -D /usr/share/wordlists/dnsmap.txt -t std --xml dnsrecon.xml
    dnsrecon.py -n ns1.example.com -d example.com -D subdomains-top1mil-5000.txt -t brt
  • nmap --script dns-brute --script-args dns-brute.domain=domain.com,dns-brute.threads=6,dns-brute.hostlist=./sub1000000.lst
  • SDBF - smart DNS bruteforcer (paper)
  • DNSenum
  • gobuster - tool for URL and DNS bruteforce
  • manually check existance of dev.example.com, beta.example.com, db.example.com, admin.example.com, …


e-mail harvesting


Technique works through bruteforcing bucket names and searching for public buckets.


Social engineering / phishing

The social engineering framework - a searchable information resource for people wishing to learn more about the psychological, physical and historical aspects of social engineering.

Social engineering questions: who? (clients/employees), purpose? (awareness assessment, checking Incident Response Center, get confidential information, …), intruder model (insider/outsider), when? (at night, at the end of working day, …)

  • SET - the Social-Engineer Toolkit
  • urlcrazy (kali) - tool for generating and autochecking availability of domain names with similar spelling
    dnstwist - domain name permutation engine for detecting typo squatting, phishing and corporate espionage
  • GoPhish - opensource phishing framework
    King phisher - phishing campaign toolkit
    Fierce Phish - other phishing framework (looks young)
  • evilginx2 - standalone man-in-the-middle attack framework used for phishing login credentials along with session cookies, alowing to bypass 2-factor authentication.
    evilginx 2 - next generation of phishing 2FA tokens
  • spoofbox.com - spoof e-mails, messangers, …

  • For spam delivery:

    • dnsbl.info - database of blacklisted ip-addresses


    • mailgun.com - “powerful APIs that enable you to send, receive and track email effortlessly (10,000 emails free every month)”
    • sendpulse.com - “maximizing open rates automatically with Artificial Intelligence, Hyper-personalization, Predictive analysis for email, SMS, Web Push, SMTP”
  • mail-tester.com - first send your email, then check your score

protection methods:

search for phishing sites: altdns - generates permutations, alterations and mutations of subdomains and then resolves them

  • configure domain (example.com TXT record “v=spf1 +a +mx -all”), mail-servers, spam-filters, sandboxes, etc.
  • monitor anomalies
  • employee training
  • carry socio-technical testing

phishing emails

phishing:

Specific attacks:

  • phishing urls/files:

    • IDN homoglyth attack (against Outlook 2013/2015/2016, The Bat)
    • RTLO (Right-to-left symbol)
    • outlook href file:// (leak NetNTLM)
    • file xxx.url (leak NetNTLM on double click)

  • malicious files:

    • PDF + macros

    • Word + smth

      • csv (lots of warnings)
      • CVE-2017-0199 (RTF)
      • Word OLE (Object Linking and Embedding)
      • Download from remote resource (MS Office)
      • word marcos
      • JS, MHT, HTA
      • packing into archive
      • DDE - Dynamic Data Exchange

Malicious e-mail themes:

  • very noisy email themes:

    • email from chiefs
    • emails about salary, bonus, dismissal, …

  • quiet emails:

    • undelivered email
    • сolleagues correspondence
    • internal mailings (questioning, health insurrance)
    • usual mail (orders, medical certificate, documents to sign)


Metadata

crafting metadata

  • FOCA (Fingerprinting Organizations with Collected Archives) - search for company’s documents (through google, yandex, bing, rambler, etc.) and afterwards exports and consolidate metadata (FOCA not maintained anymore, but still brilliant)
  • Belati - the traditional swiss army knife for OSINT (FOCA’s good/better alternative)

  • metagoofil - extracting metadata from public documents found by google

    metagoofil -d example.com -t pdf -l 100 -n 25 -o example -f example.com.html - scan for documents from a domain (-d example.org) which are PDF files (-t pdf), searching 100 results (-l 100), download 25 files (-n 25), saving the downloads to a directory (-o example) and saving the output to a file (-f example.com.html)

  • snitch - automate information gathering process for specified domain


exiftool -jk - tool for extracting metadata from files

analyzing metadata

Metadata can be treated as bigdata: splunk (offitial site)

Articles:

Tricks

  • email headers may contain ip-addresses from internal companie’s infrastructure


Other approaches

  • Lookup github.com, bitbucket.org and other open control version systems for client’s backups, configs, dev code, etc.
    GitMiner - tool for advanced mining for content on Github


Resources