PacNOG: Seventh PacNOG Meeting, Conference and Educational Workshop: Track 2 Course Web Site
PacNOG Mirror Site (US)
Kindly hosted by NSRC
PacNOG Main Page
PacNOG 7 Home

PacNOG 7: Track 2 Course Web Site

Core Topics

  • Linux OS, System Administration and IP Fundamentals
  • Configuring and Securing LAMP, Creating a Secure LAMP application, IP Services & Security
  • DNS Infrastructure, Robustness & Reliability & Best Practices DNSSEC


To raise the overall level of system administration expertise for participants. This includes methodologies for securing hosts running IP-based services, securing popular hosted services such as LAMP and DNS, and understanding the technology behind DNSSEC and how a zone is signed.


  • Track 2: Tuesday, June 29th - Saturday, July 3rd
  • Linux/UNIX Introduction Course: Sunday, June 27th.


Participants are expected to have a basic understanding of Linux or UNIX operating systems, including how to use the command line, edit files in a console-based editor (vi, nano, pico, etc.) and how to move around the Linux/UNIX file system.

Participants who feel they may need a refresher in these areas are encouraged to attend the half-day Linux/UNIX System Introduction Course to be held at the Tradewinds hotel in Pago Pago on Sunday, June 27th.


Configuration Files

Are located in configs/. These are the files we used to build the various software packages installed during the week and on the workshop NOC box.

Network Monitoring Tools

(These were running during the workshop)

Detailed Outline


Session I
* Welcome and Logistics: [PDF] (Hervey Allen, Andy Linton)
	- Instructors
	- Schedule
	- Topics
* Working with Ubuntu Linux: [PDF | PowerPoint] (Hervey Allen)
	- Philosophy 
	- Terminology
	- Packaging system
		+ Meta packages
	- Keeping up-to-date


Session II-III
* System Administration: [PDF | PowerPoint] (Hervey Allen)
	- Terminal and Console Access: [PDF | PowerPoint]
		+ OOB Access
	- Run levels
	- Configuring multiple network interfaces
	- Backup strategies: [PDF | PowerPoint]
        - RAID
	- Deploying Multiple Systems
	- Monitoring
	- SSH - A Critical Tool

LABS: Console Exercises: [PDF]
      Backup Exercises:  [PDF]
      System Admin Exercises [TEXT]

Session IV
* IP Refresher session: [PDF | PowerPoint] (Andy Linton)
	- OSI Layers
	- Netmask calculation
		+ Gateway, Nework
	- Routing basics
		+ Default vs. Static
		+ Aggregation (supernetting)



Session I
* IP Refresher and Terminology Cont.: [PDF |PowerPoint] (Andy Linton)
	- Terminology
		+ bps, pps, rtt, jitter, 
		+ packet loss
		+ flow control, congestion
	- Common tools
		+ netstat, lsof, tcpdump, wireshark, nmap, iperf

Visual tools that produce results like netstat, vmstat, ping, etc:


Session II
* Host Security Overview: [PDF | PowerPoint] (Andy Linton)
	- Onion concept of security
	- How secure do you need to be
	- Tradeoffs
	- More security = 
		+ harder access
		+ more work
	- Installation methods (off-net)
	- Immediate IDS (fcheck/tripwire) install and configuration
	- Verify what services are running
		* netstat or lsof
		* nmap for remote check
	- Turn off unnecessary services
	- Lock-down running services
		+ ssh
			* several approaches
			* keys only
			* no root access with pw
			* restrict users
			* protocol 2 only
			* changing default port
		+ Apache (if/when installed)
			* Will be covered later
		+ ftp
			* No account access (passwords in the clear)
			* anonymous logins OK. Consider isolating upload dir.
		+ Others

LAB: [TEXT] (Exercise 1)
        - Decisions:
                + give root a pw? (Debian specific)
                + use sudo or not?
        - ssh config (ubuntu, root can login bad)
                + make config changes, use reload
                + test you can actually connect, then commit
        - Running services (ps)
        - Accessible on the net (netstat lsof)

Session III
Host Security Overview cont.: [PDF | PowerPoint] (Andy Linton)
	- Logging
		+ Critical and often overlooked.
		+ syslog-ng vs. syslog
		+ swatch
	- User security
		+ strong passwords
		+ is shell access necessary
		+ are user accounts necessary (virtual mailboxes)
		+ quotas or isolate /home
	- filesystem security
		+ /tmp as nodev,nosuid,noexec
		+ /var as a separate partition


* Securing our hosts lab
	- Some steps will be done in the LAMP lab.
	- Verify services that are running
	- Turn off unnecessary services
	- Secure ssh
	- Use fcheck for IDS
	- Update /etc/fstab for /tmp
	- Configure proper logging
	- Verify user password algorithms (what's in use?)
		+ PAM configuration
	- Consider quotas
	- Note partitioning scheme in place
	- Use nmap to scan host remotely
	- Run chkrootkit kit
	- Make sure your host is monitored (to be done by NOC)
        - What do you do about CMS?
                + Drupal
                + Joomla
                + Wordpress
        - Using virtual machines
        - User jails
        - Application jails/chroots

Session IV
* Cryptography Overview: [PDF | PowerPoint] (Hervey Allen)
         - History
         - Terminology
                + Symmetric Ciphers
                + Public/Private key systems
                + Hybrid systems
                + Hashing functions
                    * hands-on sha1sum exercise
                + Passphrases
                + Digital signatures
                + Digital certificates
         - Man in the Middle Attacks
         - TLS/SSL
         - PGP and PGP trust model

         - Munge a document
         - Generate public/private ssh key pair
         - Copy your public key to another account on a remote machine


Session I
* Firewalls: [PDF | PowerPoint] (Hervey Allen)

	- Firewalls - are they necessary on a server?
		+ Yes/No
		+ problems (most attack vectors aimed at open services)
		+ false sense of security
		+ Useful to protect others from _you_
		+ Useful (in some cases) in case of DDoS attacks
			* Your provider is your real defense
	- iptables basics
		+ Installing iptables tools in Ubuntu
		+ A very simple ruleset
		+ Flushing, saving rules
		+ Verifying that rules start at system boot


Session II
* Configuring Ubuntu for use with LAMP: [PDF | PowerPoint] (Hervey Allen)
	- Package vs. Source install (security considerations)
	- Install Apache, MySQL, PHP
	- Secure Apache, MySQL, PHP


Session III
* Securing LAMP-specific items: [PDF | PowerPoint] (Hervey Allen)
	- Possible Apache security items
		+ Access controls
		+ mod_security
		+ Use virtual hosts
		+ run chroot'ed apache
		+ use ".html" for php files
		+ Options in access controls
		+ Restrict access to files.
		+ Use SecFilter options	
			* ScanPOST
			* Selective HTTP
			* Selective OUTPUT
			* ForceByteRange
		+ Hide php errors
		+ Using ssl (to be covered in detail a bit later)
	- PHP Security
		+ expose_php = Off
		+ display_errors = Off
		+ Logging
	- MySQL Security 
		+ Set root password
		+ Rename admin account from root
		+ remove anonymous accounts
		+ remove test dbs if installed
		+ bind mysql to listen on localhost only


Session IV


* Building secure LAMP application(s): [PDF | PowerPoint] (Hervey Allen)
	- Use URL encoding
	- Verify (don't trust) user input
	- Avoid global variables
	- Force redirect to https for sensitive information
		+ Using php directives
	- Create a MySQL database
	- Create a php form to create new user
		+ Captures user input
		+ Updates (writes to) MySQL db
	- Display form results
		+ Reads MySQL db


Session I-II
* Introduction to DNS: [PDF | PowerPoint] (Andy Linton)
    (These slides come from the excellent series provided by Herbert Haas at
        - IP addresses vs. names
	- DNS as a distributed, hierarchical database
	- Domain names and resource records:
	        + A, PTR, MX, CNAME, TXT, SOA/NS
	- Domain name lookup responses
	- Reverse DNS
	- DNS as client-server model
	       + Resolver
	       + Cache
	       + Authoritative server
        - Authoritative servers
               + Master/Slave configuration
               + Zone transfers
	- Resource record NS: referral of answer
	- Caching nameserver and root servers
	- Caching used to reduce load (esp. top level servers)
	- Issue of stale data in caches (problems with distributed systems).
	       + TTL records on each record
	       + Negative TTL in SOA
	- Recursion and caching (dig +norec)

* DNS Debugging lab: [HTML]
        - using 'dig' to debug DNS servers
		+ reply sections
        - options: +norec, +trace, +edns=0, +bufzize=, +dnssec
        - flags: AA, RD, RA, DF, DO/AD,
        - issues: packet size/replies, TCP/UDP, DF bit
		+ following a delegation
        - using 'doc' to debug zones and delegations
        - tcpdump and wireshark
        - packet capture
        - analysis on screen

Session III-IV
* DNS Delegations: [PDF | PowerPoint] (Andy Linton)
        - Zone replication
        - Zone format
        - Delegation
        - Common mistakes
        - Dealing with V6 and DNS (AAAA)

        - Authoritative DNS & delegation [PDF]
        - More delegation [PDF]
         - Goal: to properly configure an authoritative nameserver	
	         + Recap of caching NS	
	         + DNS Replication 
	         + Outside world cannot tell the difference between master and slave 
	         + When does replication take place? 
	         + Two (2) Dangers with serial numbers 
	         + Configuration of Master & Slave NS 
	                - Format of Resource Records { SOA and NS } 
	 - Ten (10) Common DNS Operational and Configuration Errors (RFC1912) 


Session I
* Public/Private Key Refresher: [PDF | PowerPoint]
* DNSSEC Introduction: [PDF | PowerPoint] (Hervey Allen)
         - What are we trying to solve?
                + Cache poisoning attacks 
                + Verifiable response (authenticity and integrity of response)
         - What DNSSEC does not solve
                + Authorization
                + Confidentiality of data
         - Provides independent trust path (remember PGP vs. CSA's?)
		 - Discussion of new DNSSEC records:
		 	    + DNSKEY
		 	    + RRSIG
		 	    + NSEC/NSEC3
		 	    + DS
		 - Additional DNSSEC resources available [here].

Session II
* Issues with DNSSEC: [HTML] (Andy Linton)
* Closing, Q&A

Closing Ceremony
* Certificates

  Last modified: Mon Jun 28 16:41:38 SST 2010