open-source intelligence (OSINT - wikipedia)

The Pyramid of Pain
Knowlesys - OSINT realization - looks like resource which describes osint in general

Internet is based on:

  1. Hierarchy of DNS names (tree hierarchy)
  2. RIPE databases - exists 5 regions (Europe, Central Asis; North America; Asia, Pacific; Latin America, Caribbean; Africa) each region has its own ip-address pools and each region gives sub-pools to other instances (company or provider or country or …)
  3. Set of autonomous systems - AS. (these has no hierarchy)
  4. SSL certificate chains

Metadata concept

  • by what? the file was created/changed - software type (e.g. MSWord, ImageMagick, …)
  • by whom? the file was created/changed - username, impersonalization
  • computer name, where file was created/changed
  • when? the file was created/changed - date/time
  • where? the file was located - path disclosure
  • e-mail addresses
  • ip-addresses
  • dns-names and subdomains

Most popular assests searched for compromisation:

  • an unpatched server connected to the Internet
  • an individual



  • OSINT Framework - awesome collection of various tools for OSINT (Open Source Intelligence)

  • publicwww - find any alphanumeric snippet, signature or keyword in the web pages HTML, JS and CSS code
  • - quality leads from all over the web

OSINT multifunctional tools / frameworks

  • only subdomain enum:

    • Sublist3r - fast subdomains enumeration tool for penetration testers - aggregates output from lots of sources (google,, bing, virustotal, …)
      python -d - passive
      python -b -v -d - active
    • subfinder (passive) - a subdomain discovery tool that discovers valid subdomains for websites
      better use docker
    • censys-subdomain-finder (passive) - enumeration using the crt logs (
      python --censys-api-id [API_ID] --censys-api-secret [API_SECRET]
      censys-enumeration (passive) - a script to extract subdomains/emails for a given domain using SSL/TLS certificates dataset on Censys (json output)
      python --verbose --subdomains --emails domains.txt
    • amass (passive with dns or active) - in-depth subdomain enumeration
      purely passive: ... -nodns ...
      passive: amass -v -ip -min-for-recursive 3 -log ~/amass.log -d,
      has active methods: -active -brute
    • knockpy (active) - subdomain scan
    • (passive + bruteforce) - automation of recon-ng subdomain discovery
      ./ -a

    Not all available technics are used by these tools, e.g. you can check specific technics from subdomain enumerate category (e.g. CSP analysis for subdomain search)

  • full-featured tools:

    • aquatone - a tool for domain flyovers
      Add keys: aquatone-discover --set-key [censys_id, censys_secret, shodan, passivetotal_key, passivetotal_secret, virustotal, riddler_username, riddler_password] [VALUE]
      aquatone-discover --domain --threads 25 - subdomain enumeration
      aquatone-scan --domain --ports large - enumeration common ports, used for web-services
      aquatone-gather --domain - retrieve and save HTTP response headers and make screenshots
      aquatone-takeover --domain - check subdomain-takeover situations
    • datasploit (passive + active) - osint + active scans = HTML report
      datasploit -d

    • fast analysis

      • domain_analyzer - search all info about domain
      • domain-profiler - a tool that uses information from whois, DNS, SSL, ASN, …
      • lazyrecon (active) - sublist3r and certspotter + screenshots + grab response header + nmap + dirsearch = generate HTML report
  • theHarvester (passive + active) - e-mail, subdomain and people names harvester
    python -b all -d
  • DMitry (active + port scan) - gather as much information as possible about a host. Base functionality is able to gather possible subdomains, email addresses, uptime information, tcp port scan, whois lookups, …
    dmitry -i -w -n -s -e
    with port scan: dmitry -i -w -n -s -e -p -b -t 2

  • web-spidering:

    • BlackWidow - web-spider
      /usr/share/BlackWidow/blackwidow -d -l 5
    • Photon - light web-spider -u -l 5 -d 0 -t 10
    • blacksheepwall (based on CommonCrawl - grep the internet)
      blacksheepwall -cmn-crawl CC-MAIN-2018-13-index -domain

  • - official site, github
  • spiderfoot – open source intelligence automation tool for process of gathering intelligence about a given target, which may be an IP address, domain name, hostname or network subnet
  • recon-ng (kali linux) - good (and huge) tool for various reconnaissance vectors

    usage sample
    workspaces; workspaces add
    show modules
    # certificate transparency query:
    use recon/domains-hosts/certificate_transparency; show info; set SOURCE;
    # netcraft
    use netcraft; set source;
    # Resolve Hosts, get IPs, GEO and report:
    show hosts
    use resolve; use recon/hosts-hosts/resolve; run
    use freegeoip; run
    use report; use reporting/xlsx; run

scanning tools

  • subresolve - resolve and quickly portscan a list of sub-domains

Subdomain / ip / e-mail harvesting / enumirate / etc. (concrete tools)

Subdomain enumiration
process of exposing subdomains of one or more domains

network recon

subdomain recon

Categorial/concrete tools/attacks:

  • CloudFail - utilize misconfigured DNS and old database records to find hidden IP’s behind the CloudFlare network
    python --target

subdomain enumerate

  • domains-from-csp - a script to extract domain names from Content Security Policy(CSP) headers
    python -r

Everything beneath can be done faster if you will use frameworks and other complex tools

  • Subject Alternative Name (SAN) - X509 extension to provide different names of the subject (subdomains) in one certificate

    Even if there is non-resolvable subdomain, probably admins use the same certificate for intranet connections.

  • Forward DNS

  • zone transfer - does DNS server expose a full DNS zone? (via AFXR) (AXFR zone transfer scan (by sergeybelove))

    dig axfr
    host -t axfr
    host -avl
    nslookup -query=AXFR
    • fierce -dns
    • dnsrecon -a -d
  • NSEC walking attack - enumerates DNSSEC-signed zones
    Take your DNSSEC with a grain of salt

    • apt-get install ldnsutils

      • ldns-stroll
      • ldns-walk
    • nsec3map – DNSSEC Zone Enumerator – позволяет перебрать содержимое всей доменной зоны и найти поддоменты, если на dns сервере работает dnssec (
    • nsec3walker
    • nmap -sSU -p 53 --script dns-nsec-enum --script-args <target>
      nmap -sSU -p 53 --script dns-nsec3-enum --script-args <target>

subdomain bruteforce

Comparison of subdomain bruteforce tools: massdns, gobuster, dns-paraller-prober, blacksheepwall, subbrute (pic)
SecLists - check bruteforce lists
compiled GIANT subdomain wordlist (march 2018)

  • massdns
  • fierce
    fierce -dns
    fierce -dns -wordlist /path/to/wordlist.txt - for custom wordlist
  • subbrute
  • dnsrecon
    dnsrecon -d -D /usr/share/wordlists/dnsmap.txt -t std --xml dnsrecon.xml -n -d -D subdomains-top1mil-5000.txt -t brt
  • nmap --script dns-brute --script-args,dns-brute.threads=6,dns-brute.hostlist=./sub1000000.lst
  • SDBF - smart DNS bruteforcer (paper)
  • DNSenum
  • gobuster - tool for URL and DNS bruteforce
  • manually check existance of,,,, …

e-mail harvesting

Technique works through bruteforcing bucket names and searching for public buckets.

Social engineering / phishing

The social engineering framework - a searchable information resource for people wishing to learn more about the psychological, physical and historical aspects of social engineering.

Social engineering questions: who? (clients/employees), purpose? (awareness assessment, checking Incident Response Center, get confidential information, …), intruder model (insider/outsider), when? (at night, at the end of working day, …)

  • SET - the Social-Engineer Toolkit
  • urlcrazy (kali) - tool for generating and autochecking availability of domain names with similar spelling
    dnstwist - domain name permutation engine for detecting typo squatting, phishing and corporate espionage
  • GoPhish - opensource phishing framework
    King phisher - phishing campaign toolkit
    Fierce Phish - other phishing framework (looks young)
  • evilginx2 - standalone man-in-the-middle attack framework used for phishing login credentials along with session cookies, alowing to bypass 2-factor authentication.
    evilginx 2 - next generation of phishing 2FA tokens
  • - spoof e-mails, messangers, …
  • For spam delivery:

    • - database of blacklisted ip-addresses

    • - “powerful APIs that enable you to send, receive and track email effortlessly (10,000 emails free every month)”
    • - “maximizing open rates automatically with Artificial Intelligence, Hyper-personalization, Predictive analysis for email, SMS, Web Push, SMTP”
  • - first send your email, then check your score

protection methods:

search for phishing sites: altdns - generates permutations, alterations and mutations of subdomains and then resolves them

  • configure domain ( TXT record “v=spf1 +a +mx -all”), mail-servers, spam-filters, sandboxes, etc.
  • monitor anomalies
  • employee training
  • carry socio-technical testing

phishing emails


Specific attacks:

  • phishing urls/files:

    • IDN homoglyth attack (against Outlook 2013/2015/2016, The Bat)
    • RTLO (Right-to-left symbol)
    • outlook href file:// (leak NetNTLM)
    • file xxx.url (leak NetNTLM on double click)
  • malicious files:

    • PDF + macros
    • Word + smth

      • csv (lots of warnings)
      • CVE-2017-0199 (RTF)
      • Word OLE (Object Linking and Embedding)
      • Download from remote resource (MS Office)
      • word marcos
      • JS, MHT, HTA
      • packing into archive
      • DDE - Dynamic Data Exchange

Malicious e-mail themes:

  • very noisy email themes:

    • email from chiefs
    • emails about salary, bonus, dismissal, …
  • quiet emails:

    • undelivered email
    • сolleagues correspondence
    • internal mailings (questioning, health insurrance)
    • usual mail (orders, medical certificate, documents to sign)


crafting metadata

  • FOCA (Fingerprinting Organizations with Collected Archives) - search for company’s documents (through google, yandex, bing, rambler, etc.) and afterwards exports and consolidate metadata (FOCA not maintained anymore, but still brilliant)
  • Belati - the traditional swiss army knife for OSINT (FOCA’s good/better alternative)
  • metagoofil - extracting metadata from public documents found by google

    metagoofil -d -t pdf -l 100 -n 25 -o example -f - scan for documents from a domain (-d which are PDF files (-t pdf), searching 100 results (-l 100), download 25 files (-n 25), saving the downloads to a directory (-o example) and saving the output to a file (-f

  • snitch - automate information gathering process for specified domain

exiftool -jk - tool for extracting metadata from files

analyzing metadata

Metadata can be treated as bigdata: splunk (offitial site)



  • email headers may contain ip-addresses from internal companie’s infrastructure

Other approaches

  • Lookup, and other open control version systems for client’s backups, configs, dev code, etc.
    GitMiner - tool for advanced mining for content on Github