Squid, SquidGuard, and Lightsquid - pfSense Hangout March 2014

Squid, SquidGuard, and Lightsquid
Jim Pingle
ESF, LLC
March 2014 Hangout

What are Squid, SquidGuard, and LightSquid?
● Squid is a caching proxy for HTTP and other protocols
– Can speed up access by locally caching commonly loaded sites/objects
– Can save bandwidth by reducing multiple duplicate downloads
– Allows further action on web traffic (access control, reporting)
● SquidGuard is used for access control based on the URL requested by
a client
– Decisions can be made to allow or deny access based on client or destination
– Blocked sites can be redirected to an error page in most cases
– Custom lists of sites or preset blacklists from other sources
● Lightsquid is used for reporting web access history
– Parses squid's log, notes who went where, how much bandwidth they used,
and other factors
– Has reports for daily use, monthly, and so on
– Does not work for NanoBSD

Will Squid work for me?
● Before starting, determine if Squid suits the situation
● Without squid, you cannot take an action based on the contents of HTTP
packets.
– In pf, all that may be seen is the destination IP and port number, not hostname
– Hostnames can resolve to many IP addresses, random sets, cannot be effectively
tracked with aliases in many cases.
– As a proxy, squid sees the whole HTTP transaction including the requested site name
● Squid can transparently capture HTTP, not HTTPS
● In future versions HTTPS capturing is possible but still requires installing a trusted root CA on
clients
● HTTPS can be handled by manual proxy configuration, or via WPAD or similar
● Squid is not easily compatible with Multi-WAN
● SquidGuard tests based on client and destination URL, not page content
● Squid can help by caching static content, but many pages are dynamic and
may not cache well or at all so the savings may not significant
● On NanoBSD, it can only be used for access control, not caching

What will be covered?
● Squid 2.7.x + SquidGuard - Proven stable configuration
● Reporting with Lightsquid, sqstat
● Transparent proxy of HTTP traffic
● Manual browser configuration for HTTP/HTTPS
● Blocking sites with SquidGuard
● Blacklists and Target Categories
● Custom ACLs
● Error messages and redirects

What will NOT be covered?
● WPAD, Proxy Auto Configure
● Moving Squid cache to an additional disk
● SquidGuard Schedules
● External authentication sources such as LDAP with SquidGuard
● Squid 3.x
● Sarg
● Anything related to Multi-WAN – Squid traffic will always take
the default gateway
● Interactions with Captive Portal
● Secondary/Upstream Caches
● HAVP

Prerequisites for Setup
● For the purposes of this presentation...
● Firewall VM – Clean config, no packages.
● Client VM – Basic client, just an OS and a web
browser
● Ensure you have working DNS and routing on the
firewall
● Install packages from System > Packages on the
Available Packages tab
● Install Squid (2.7.x!)
● Install SquidGuard (Not -devel or -squid3!)
● Install Lightsquid

Configure Squid – Main Tab
● Squid Settings are located under Services > Proxy Server
● Proxy Interface - Select the local interface, e.g. LAN/LAN2/Wifi –
Sets squid to Listen
● Allow users on interface - check – automatically adds ACLs
● Transparent Proxy - check to intercept HTTP requests
● Bypass options
– Source is for client IP addresses
– Destination is for remote IP addresses (servers)
– Do not use hostnames here. Use aliases if you must use hostnames
● Logging (only required for reporting), rotation
– Not viable for NanoBSD
● Proxy port - leave at 3128

Configure Squid – Main Tab (cont'd)
●
hostname and e-mail - for error messages
● Disable X-Forward
– Does not disclose the true internal client IP to the remote server.
– Setting this is best for privacy.
●
Disable Via
– Hides the fact that a proxy is involved, for security, privacy, or technical reasons.
– If you did not also disable X-Forward, a web server can still detect a proxy!
● Suppress Squid Version
– Hides the squid version in the Via HTTP headers, for security, privacy, or technical reasons.
– If Disable Via is checked, this setting does not matter as the entire squid Via header is
omitted.
● Custom Options
– Manually enter your own squid directives
– SquidGuard and others may put settings here
– If you don't know what they are, they are best left as-is

Configure Squid – Cache Mgmt Tab
● Hard disk cache size - in MB, set to 0 on NanoBSD
– Be careful with large caches. On 32-bit, Squid takes about 10MBytes of RAM
per 1GB of cache. On 64-bit, it takes 14MBytes of RAM per 1GB of cache.
● Hard disk cache system - ufs, or null for NanoBSD
● Hard disk cache location - directory, defaults to /var/squid/cache
● Memory cache size – This does not specify a maximum for squid, just
for caching objects in RAM. Can be exceeded if required for a specific
request.
● Object sizes
– Minimum Object Size: Smaller than this not saved to disk
– Maximum Object Size: Larger than this not saved to disk. Speed = Low,
Bandwidth saving = High
– Maximum Object Size in RAM: Smaller than this will be put in memory cache.

Squid – Access Control Tab
● Allowed Subnets - typically not needed when
using "allow users on interface" but if needed,
enter (Additional) subnets here
● Banned hosts, whitelist, blacklist - not typically
used with SquidGuard
● External Cache Managers - make sure LAN IP
is listed or sqstat can fail with 403 error
– If sqstat will be accessed from another interface,
add it here too (e.g. WAN IP, Management LAN IP)

Squid – Traffic Mgmt Tab
● Sizes are in Kilo*bytes*, not bits – Important to remember for the
throttling settings. Set to 0 to disable limits.
● Max down/up sizes – Careful, can break updates
● Throttling – Overall and per-host
● Throttle extensions
– Binary Files: bin, cab, sea, ar, arj, tar, tgz, gz, tbz, bz2, zip, 7z, exe,
com
– CD images: iso, bin, mds, nrg, gho, bwt, b5t, pqi
– Multimedia: aif/aiff, asf, avi, divx, mov, mp3, mp4, wmv, mpg/mpeg, qt,
ram/rm
– Other extensions
●
Custom list, just put in the letters in the extension
● NO spaces between items!
● Example: vmx,vdi,ova

Squid – Auth Settings tab
● Not used with transparent mode, but can be
setup to use local authentication, LDAP,
RADIUS, or NT Domain login
Squid settings complete! On to SquidGuard

SquidGuard
● SquidGuard Settings may be found at Services
> Proxy Filter
● SquidGuard is used for allowing or denying
based on the URL requested by the client (e.g.
domain name, part of URL) and the client itself
● Currently does NOT filter based on text inside
page

SquidGuard – General Tab
● Check the box to Enable
● ALWAYS RETURN HERE AND APPLY AFTER
CHANGES!!
● Logging, enable/rotate
● Clean advertising
– Looks for blacklists with _ads or _adv in name, URLs in these
lists get replaced with blank image
● Blacklists
– MESD – Free to use by all: http://squidguard.mesd.k12.or.us/
– Shalla – Free for non-commercial use: http://www.shallalist.de/
– Others: http://www.squidguard.org/blacklists.html

SquidGuard – Blacklist Tab
● Used for downloading the blacklist archive
defined on the General Settings tab
● Progress is displayed while downloading the list
and updating the databases
● Database update can take a long time
especially on slow hardware or systems with
slow disks

SquidGuard – Define Categories
● Categories can be defined manually, using
blacklists, or both
● For blacklists, define the blacklist archive URL
on the general tab and then use the Blacklist
tab to download the blacklist
● For custom categories, use the "Target
Categories" tab

SquidGuard – Target Categories
●
Custom lists to block or pass
● Name – Name for this Target Category
● Ordering
– Important when making whitelists, reorder those to the top when possible
● Domain list
– Most common
– Will block subdomains also. For example "facebook.com" also blocks "apps.facebook.com"
– Does not block other domains that overlap in name, for example "facebook.com" does not
block "ihatefacebook.com"
● URL List
– Similar to domain list but matches both domain and URL
– Example: google.com/maps/
● Regular Expression
– Matches portions of a URL based on regex patterns.
– Example from the GUI: mail|casino|game|.rsdf$
● Redirect Mode/Redirect – Will cover that under Common ACL

SquidGuard – Common ACL
●
Default category actions for everyone that is not matched by an ACL
● Each category appears in the list, both custom and from blacklists
● Access types:
– “---” - No action taken for this later category
– allow – Allow if never blocked
●
If the site is in a later category that is Denied, it will still be blocked!
– whitelist – Allow even if blocked
– deny – Do not allow access
● To deny access to all web sites by default, set "Default access [all]" to "deny".
Other lists would then generally be used as whitelists of approved sites.
● Do not allow IP Addresses in URL
– Will not let users connect to http://x.x.x.x/ forcing the use of hostnames instead.
– Some sites will load properly if you address them that way, and a user could bypass the
protections in place.
● Proxy Denied Error
● If using the internal error page, this text will be placed at the top of the generated
error page.

SquidGuard – Common ACL (cont'd)
● Redirect Mode:
– Select the type and enter a proper value in the box if needed.
– none - Deny content without displaying a formal error.
– Internal Error Page: Enter an error message in the box, displayed with other blocked page details
– Internal Blank Page: Returns an HTML page, but blank
– Internal Blank Image: Returns an image file, but blank (Useful for Ad blocking)
– External URL Error Page
●
Appears to the browser as the actual requested page
●
Cannot include other files unless they are inline in the html (no external css, images, etc)
●
Automatically has SquidGuard variables appended to pass the requested URL, client IP/name, group, target to the script
– External URL Redirect: Same as "External URL Error Page" except it does not append the request variables
– External URL Move - Redirects with 301
● Client browser is redirected to an external error page but in a way the client browser knows that it happens (301)
● 301 is a permanent redirect, and browsers will often cache it as such
● Since it is an external page, it can use any external resources you like
●
Does not get the request variables appended, but you can add them yourself (?a=%a&n=%n&i=%i&s=%s&t=%t&u=%u)
●
Because it is an external page in a separate request, your ACLs in squidGuard must allow access to the page if it is remote
– External URL Found - Redirects with 302
● Same as above but uses a 302 response
● 302 is temporary so the browser will keep trying the original URL for later requests
● Use SafeSearch engine
– For known search engines, forces the use of the "safesearch" mechanism to prevent loading of adult material

SquidGuard – Group ACL Tab
● Similar to Common ACL but applies to only a specific set of users
● Name – Custom name of the ACL (e.g. user's name, workgroup, set of users)
● Order – Make sure to list more specific ACLs (e.g. 192.168.1.5) ahead of more
general ones (192.168.1.0/24) so that your desired actions will take place
● Client (source)
– Entries separated by space
– IP adresses, subnets, IP Ranges, hostnames, or usernames (if using user auth without
transparent mode)
● Time – Schedule used for "off-time" decisions. Not covered in this presentation.
● Target Rules
– Same mechanism as Common ACL
– Must specify an action for ALL categories for which an action should be taken
– Does NOT “fall through” to common ACL settings
– Left column is used with or without schedules
– If a time is chosen, left column is inside the time period, right column is outside
● Other options are the same as Common ACL

SquidGuard – Times Tab
● Defines schedules for use with ACLs
● Examples: WorkTime, LunchHour, Weekend,
WorkWeek
● Not covering details of this today

SquidGuard – Log tab
● For viewing squidguard logs generated by the GUI and other
actions
● Blocked: Blocked sites, if logged
● Filter GUI Log: Messages about the squidGuard GUI
configuration
● Filter Log: SquidGuard process events
● Proxy Config: View of the squid configuration file
● Filter Config: View of the squidGuard configuration file
SquidGuard Configuration Complete! On to Lightsquid

Configure Lightsquid
●
Lightsquid is used for reporting user activity logged by the squid proxy
● Lightsquid is located at Status > Proxy Report
● Enable logging in squid first!
● Language: Language for the report (e.g. English)
● Bar Color: Color for bars in graphs
●
Report Scheme: Theme to use for the report (look/feel)
● IP Resolve Method
– Used for future parsing, will not reparse old records
– IP - IP Address only, no parsing
– Demo - Authname if known, then DNS if known, then IP
– DNS - Resolves by reverse DNS if possible
●
Useful if registering DHCP in DNS and using the right DNS server, or if you have another local DNS
server
– Simple - Authname if known, otherwise IP
– SMB - SMB name of PC (resolves via NetBIOS?)
– Squidauth - Authname from squid, otherwise IP

Configure Lightsquid (cont'd)
● Refresh Scheduler
– Sets up a cron job to automatically reparse squid
access log and update the report
– Depending on the speed of the box and the activity in
the logs, 30m-60m are good times, or less often if the
report is viewed infrequently.
● Refresh Now: Update the report now
● Refresh Full: Generate the full report again now
(best to use this when making the first report)
● Skip URL: List of URLs to not report (e.g.
monitoring systems, internal pages, etc)

Lightsquid Reports
● Status > Proxy Report, Lightsquid Report tab
● Access report by year, month, day
– Most common to click on Day in main report
● Top sites list
– Can be sorted by Bytes or Connections
● Totals
– Total bandwidth used by a client over all time
● Day report shows users that made connections on that
specific day
– Click on IP in the report to get a list of sites they visited
● "Big Files"
– List of files larger than a few MB and who downloaded those files

Lightsquid – Proxy State (sqstat)
● RealTime view of squid downloads
● NOT a live view of the access log
● Error (1): Cannot get data. Server answered:
HTTP/1.0 403 Forbidden
– Fix by adding IPs on the firewall to squid's Access
Control tab, "External Cache-Managers"
– IP to add would be the one you reach the GUI for
management. If you come across LAN, use LAN IP. If
managing remotely, use WAN IP(s).
Lightsquid Complete! Now for testing...

Browser Tests
●
Use Incognito Mode, Private Browsing Mode, etc. Avoid issues with cached
redirects.
● Load a proxy test page such as http://www.lagado.com/proxy-test
●
Load a normal unblocked/whitelisted web page
●
Load a blocked web page - internal error (e.g. www.facebook.com)
●
Load a blocked web page - external error (e.g. www.bing.com)
●
Load HTTPS version of blocked page – oops, still works
● Configure browser for proxy (varies by OS/Browser)
– Set to use the IP address of the firewall on port 3128 as proxy for all
● Load blocked HTTPS page again, now it doesn't load
● Load a normal HTTPS page, it loads OK
●
Can block 443 on LAN rules if desired
●
After loading some sites, manually refresh lightsquid and then load the report
again to see if reporting works

Other Notes
● Caching Windows Updates and AV updates
– Mixed success. See the article on the doc wiki for refresh pattern
recommendations
● Swap State File
– Not generally a problem now, but in some rare cases can grow
large/fill up FS.
● Squid randomly dies w/Signal 6
– If Squid stops working after a reboot and seems to die randomly, or
runs extremely slowly, odds are that the cache is corrupt
– Run:
● mv /var/squid/cache /var/squid/cache.old
● squid -z
● rm -rf /var/squid/cache.old
● Restart squid in GUI while the rm runs. Takes long time w/large cache

That's All!
● Questions – Time allowing
● Possible future revisit for some items not
covered

Squid, SquidGuard, and Lightsquid - pfSense Hangout March 2014

More Related Content

What's hot (20)

Similar to Squid, SquidGuard, and Lightsquid - pfSense Hangout March 2014 (20)

More from Netgate (20)

Recently uploaded (20)

Squid, SquidGuard, and Lightsquid - pfSense Hangout March 2014