Mac OS X Server 10.6 Snow Leopard: DNS and Network Services
Illustration by John Hersey

Start your free trial now, and begin learning software, business and creative skills—anytime, anywhere—with video instruction from recognized industry experts.

Start Your Free Trial Now

Video: Deploying a VPN

When deploying a VPN you have to think about three things right away, your network router, which contains a firewall of some sort and is performing NAT most likely, your server, and its firewall settings and your client, and its compatibility with your available technologies. The first place we need to go is the router. We access that usually through a web interface. Yours may go through a Telnet session on a command line client. They are all different. So we are just going to do this here in our web browser. Log in.
please wait ...
Watch the Online Video Course Mac OS X Server 10.6 Snow Leopard: DNS and Network Services
2h 4m Intermediate Jul 16, 2010

Viewers: in countries Watching now:

In Mac OS X Server 10.6 Snow Leopard: DNS and Network Services, instructor Sean Colins introduces the networking services available in Snow Leopard Server. This course covers setting up a DNS server to provide network resources, using firewalls to protect systems against intrusion and to route traffic, using DHCP to automatically configure network settings for computers when they join a network, and accessing a network securely via a remote VPN (virtual private network) connection. Exercise files accompany the course.

Topics include:
  • Deploying, troubleshooting, and understanding OS X 10.6 DNS server
  • Understanding and configuring OS X and OS X Server-based firewalls
  • Fixing server- and client-side firewalls
  • Configuring and troubleshooting DHCP
  • Setting up and troubleshooting a VPN server
Business Developer IT
Mac OS X Server
Sean Colins

Deploying a VPN

When deploying a VPN you have to think about three things right away, your network router, which contains a firewall of some sort and is performing NAT most likely, your server, and its firewall settings and your client, and its compatibility with your available technologies. The first place we need to go is the router. We access that usually through a web interface. Yours may go through a Telnet session on a command line client. They are all different. So we are just going to do this here in our web browser. Log in.

Usually this is going to be under some sort of an advanced configuration area. In ours this is under Advanced and then Firewall Settings over here in the sidebar. Your router must support passing VPN traffic through to a VPN server. This mean that the router must support something called ESP, which is not Extrasensory Perception, but that will be funny. It's actually Encapsulating Security Payload Protocol, and it has to be passed for L2TP to work, which is one of the types of VPN supported by OS X Server.

Also necessary but somewhat out there is GRE or Generic Routing Encapsulation Protocol. That's necessary for PPTP to work. So our router calls all that stuff down here at the bottom ALG, or Application Level Gateway. You can see that they have labeled PPTP and IPsec VPN right here. We have got both of those already turned on. You need to find your router's analogy for this. Or if you've got this exact same thing, turn this on, because your router has to be able to pass this stuff through for the traffic to get through the router over to the server.

Now some routers will pass this automatically and not have an option to turn on or off, so you don't need to necessarily see something like this. It may be worth trying anyway. It might be turned on in the background. Just give it a shot. If your router doesn't support either of those, then you can't do VPN unless you put your OS X Server outside of your router and pass traffic through a second Ethernet port to your internal network. That would mean you'd be setting up your server as an Internet gateway and we covered that in Snow Leopard Server New Features.

Also, on your router you're going to have to port forward the necessary ports from the WAN side of the router to the IP address of your VPN server. So we are going to go over here to Advanced again, and then we are going to come over here to Port Forwarding. Now under Port Forwarding we need to add rules for L2TP and PPTP to work. So let's do that now. We are going to start by adding 1701, and that's the name. I'm just duplicating the name of the port so I know what's what here. 1701 is going over UDP.

So actually I need to take that out and come down here in 1701. Your configuration may allow you to put it in a number and then select whether or not it's TCP, UDP, or both. In this interface this whole section is one rule and if you put in TCP, it will do TCP. If you put in UDP, it will do that, and if you put in both, it will do both. You also have to configure the IP address that this stuff is going to on the inside of your network. So ours is our server, which is

So instead of 1701 if I wanted to I could say L2TP, because 1701 is one of the ports necessary for L2TP. It's really up to you. You could even do something like this where you put in 1701 and then type L2TP, if you want. It's really again up to you. It's important to make sure that you are allowing this and not denying the traffic in your Port Forwarding configuration and that you click the check box. We are going to fast-forward through this so that you don't have to wait through it, but just so you know we are doing 1701, 4500, and 500 over UDP when we want to do LT2P VPN, and we are going to do 1723 over TCP for PPTP VPN.

Now be very careful to get the port number and the protocol just right here. Once you're done, go ahead and scroll back up and save your settings. Make sure you've saved these settings before you get out of the interface. Otherwise of course they won't take effect. Once that's completed, we are going to go back into Server Admin so that we can configure our firewall to do pretty much the same thing. We are simply going to allow traffic from those ports.

Click Continue. Double-check to make sure that everything saved, and it did. So we are good. So what I am going to do now is I am going to quit Firefox, and we will now open Server Admin. Something else to consider would be deploying a router that has a VPN server embedded in it. I recommend using the VPN server built- in OS X Server instead, because your OS X Server can use your Open Directory users and passwords for access. If you use your router as the VPN server, you may have to re-enter all of the names and passwords into that device, which is time-consuming and inefficient for you, the administrator.

Once in Server Admin click on the Firewall Service, go to Settings, come over to Services, and we've got our various groups. I am going to go to any and I am going to select Allow only traffic to these ports, and what we are going to do is just sort on Description. This is a very cool trick. Let's come all the day down here and you will see that all the VPN services are neatly grouped in the same area. We are just going to click Allow on all of them and click Save. Now the reason we're doing this is your firewall will block the VPN traffic if you don't enable all of the ports in the appropriate firewall address groups that are necessary for VPN.

Now those would be ESP plus Port 1701, 4500, and 500 over UDP for L2TP and GRE plus 1723 for PPTP. If you're only using one of those protocols, don't just enable the firewall for the other just because. Be specific, but if you're going to be supporting both which may frequently be the case, go ahead and up them all. Now we don't need to open those up for the DHCP group, because the DHCP group is not going to be outside of our firewall.

But I am going to open up the VPN range in here for the group that will be on the VPN. Of course, for our Server Admin client everything is always allowed. So we are all good as far as that's concerned. To configure VPN, you'll need to start with a user account. And while you can do that in Workgroup Manager, we are going to do it in Server Preferences. When we open up Server Preferences, it may ask us to allow or authenticate access and eventually it will come up and show us our interface.

Now what we can do here is we can add our user accounts and you'll notice right here that we have not yet been set up to manage users and groups. So I am going to click the Set Up button right here and it's asking if we want to host Users and Groups on the server. We want to say yes. We are going to say OK. Now, because the Server Preferences application assumes that you'll be using an Open Directory master for your users, it's going to create that for you automatically. Since we've already got DNS setup properly, this won't be a problem.

It will be created just fine, and it won't alter our DNS settings because everything is configured properly. One Server Preferences sets up your Open Directory master, at least in 10.6.3 there is a bug where you need to restart your server at this point. So if you're at a later version where they've fixed that, you can just continue with me. If you're at 10.6.3 you may want to restart your server at this point so that you can create a user. Assuming that you've restarted, if you needed to, let's create that first user now. We will click the Plus button and we will type in the user's name.

I am just going to use me for now, and a password and verify that password. Click Create Account and it will create that user. Now, what I want to show you here is over under the Services tab you can enable any of the available services for that user and while this won't set up those services for the user if you don't have them already configured, what it will do is add your user to the access control list for that service. So we are just going to use VPN.

So I'm going to turn on VPN. So that will be putting this user into that access control. In addition to that, remember we were going to set up a group here. So we're going to have to do that as well. Right now, we have a group named Workgroup because that group is set up automatically during the initial setup of the Open Directory master here in Server Preferences. Let's go to Show All, let's go back here to Groups, and let's create a new group, and we will call this group VPN Users and Create Group. We are not going to have a File Sharing Folder or iChat Auto Buddies.

Under Members we will click Edit, put a check mark right there, unclick the Edit Membership button, and we are set. So we have our VPN users group. We have our new user, which has access to VPN. We are almost done with Server Preferences, but I am going to leave it open, because we are going to come back to this a little later on, and you will see it's pretty cool. So let's go back over into Server Admin where we were in our firewall and looking at our firewall ports and services. Before we move on to configuring the VPN, I do want to show you what the Server Preferences did.

We are going to click on the name of our server, come over here to Access and underneath Services, if we come down here and click on VPN, you can see that it added the Sean Colins user to the VPN group. This is what I meant whenever I was talking about the access controls here for the service. Let's click over here on Settings > Services and then all the way down here at the bottom, we've got VPN. Let's click on that. Click Save and then when it appears in the sidebar click on VPN and it will take you straight to Settings.

We are going to start here in L2TP. Now remember we specified what our VPN address ranges were going to be back when we were planning our firewall address groups and DHCP range. So we know that we have through 159 available for both L2TP and PPTP services. I am going to turn this on and I'm going to assign through to L2TP, and I'm going to assign under PPTP through

So that splits that roughly in half. Now, when configuring L2TP back over here, you have some decisions to make. Will you use a shared secret? Will you use Kerberos for authentication? I am going to assume you're going to use a Directory Service for authentication. You can select either MS-CHAPv2 or Kerberos for authentication. Thing is, you're not going to be able to use Kerberos unless your VPN server is sitting on an external IP address, and that has to do with the way that Kerberos needs DNS and its IP address and everything to match up properly. If your Kerberos is on an internal network, once you leave that network Kerberos isn't going to function properly.

So, you will probably end up needing to use MS-CHAPv2 in the Directory Services, and then for the IPSec Authentication here in L2TP you have the option of putting in either a shared secret, or you can use a certificate. Now we talked about creating certificates back in previous Snow Leopard Server classes. So I am going to talk to you about the shared secret and how that works. The shared secret is my personal favorite, because it's secure, and it's easy. This will probably be the most popular solution, and I recommend it highly as an option for L2TP VPN authentication, as it's very secure and easy for you to implement.

For that reason I am going to show you how to use a shared secret now. In the Shared Secret box type a relatively long sequence of numbers, letters, and characters that make up a password that will be shared between your server and the VPN clients on each machine that will be configured to connect to your server. So I am going to put one in now and I'll click Save. In our SPI analogy this was the part of the process where the black box sent a password only it knows to a black box far, far away.

You are setting that password here when you add the shared secret. Now I said we were going to go back to Server Preferences, and this is where we are going to do that. I am going to flip back to Server Preferences now. Now when you enter the Server Preferences application, you can save the L2TP VPN settings to a file right in here, which can be imported into a client system and used without ever revealing the shared secret to the person using the client-side VPN. We simply click here and save this in a place where we can find it. I am going to put it on the desktop and there is the file.

This is what you'll end up with and this is the file that you will send to the client system to be imported later. Now, if you want to configure your PPTP settings as well, we just go back here to Server Admin. Click on PPTP. Once you have the IP range in the IP boxes, you click the check box to either Allow 40-bit encryption for compatibility only or not. I recommend leaving this off unless you have a very old PPTP client that needs to connect to this.

You select whether you are using Kerberos or MS-CHAPv2. Again, I recommend the MS-CHAPv2 option here, or again you could use your RADIUS server if you have one. And once you have that configured, you click Save. Now pay attention, because there is no shared secret here. I mentioned that the shared secret is available in some types of VPN, and this is not one of them. There really isn't an option for increased security beyond the basic password configured in Workgroup Manager. That's entered at the client-side in the VPN Settings in System Preferences.

Let's click Start VPN, and now we can go configure a client to use the software. Once you've configured L2TP and PPTP, it's important to put client information into the interface and also configure logging. So let's click on Client Information next. You'll automatically have the DNS server set to your server and that's important, because you'll need to be able to pick up authentication information and locate services on the inside of the network. It's good to have this configured in this way.

So please leave it alone. Under Search Domains, you can add your own server's domain so that you can autocomplete connections easily just using the host name or the first part of the name of a resource. So, if you want to do that it would look like this. You can also add network routing definitions in order to determine what IP address and subnet masks are going to be routed privately or publicly whenever someone is connected over a VPN.

If you leave this unconfigured, however, everything will go through the VPN as long as you configure all traffic to go over the VPN from the client side, which I recommend you do. Under Logging, it's a good idea especially at the beginning when you're first turning on your service to enable verbose logging. Click Save and when you've got that finished, click Start VPN. Next, we need to go over to a client, configure it, test it, and if it doesn't work, troubleshoot it.

There are currently no FAQs about Mac OS X Server 10.6 Snow Leopard: DNS and Network Services.

Share a link to this course

What are exercise files?

Exercise files are the same files the author uses in the course. Save time by downloading the author's files instead of setting up your own files, and learn by following along with the instructor.

Can I take this course without the exercise files?

Yes! If you decide you would like the exercise files later, you can upgrade to a premium account any time.

Become a member Download sample files See plans and pricing

Please wait... please wait ...
Upgrade to get access to exercise files.

Exercise files video

How to use exercise files.

Learn by watching, listening, and doing, Exercise files are the same files the author uses in the course, so you can download them and follow along Premium memberships include access to all exercise files in the library.

Exercise files

Exercise files video

How to use exercise files.

For additional information on downloading and using exercise files, watch our instructional video or read the instructions in the FAQ .

This course includes free exercise files, so you can practice while you watch the course. To access all the exercise files in our library, become a Premium Member.

Join now Already a member? Log in

* Estimated file size

Are you sure you want to mark all the videos in this course as unwatched?

This will not affect your course history, your reports, or your certificates of completion for this course.

Mark all as unwatched Cancel


You have completed Mac OS X Server 10.6 Snow Leopard: DNS and Network Services.

Return to your organization's learning portal to continue training, or close this page.

Become a member to add this course to a playlist

Join today and get unlimited access to the entire library of video courses—and create as many playlists as you like.

Get started

Already a member ?

Exercise files

Learn by watching, listening, and doing! Exercise files are the same files the author uses in the course, so you can download them and follow along. Exercise files are available with all Premium memberships. Learn more

Get started

Already a Premium member?

Exercise files video

How to use exercise files.

Ask a question

Thanks for contacting us.
You’ll hear from our Customer Service team within 24 hours.

Please enter the text shown below:

The classic layout automatically defaults to the latest Flash Player.

To choose a different player, hold the cursor over your name at the top right of any page and choose Site preferences from the dropdown menu.

Continue to classic layout Stay on new layout
Exercise files

Access exercise files from a button right under the course name.

Mark videos as unwatched

Remove icons showing you already watched videos if you want to start over.

Control your viewing experience

Make the video wide, narrow, full-screen, or pop the player out of the page into its own window.

Interactive transcripts

Click on text in the transcript to jump to that spot in the video. As the video plays, the relevant spot in the transcript will be highlighted.

Learn more, save more. Upgrade today!

Get our Annual Premium Membership at our best savings yet.

Upgrade to our Annual Premium Membership today and get even more value from your subscription:

“In a way, I feel like you are rooting for me. Like you are really invested in my experience, and want me to get as much out of these courses as possible this is the best place to start on your journey to learning new material.”— Nadine H.

Thanks for signing up.

We’ll send you a confirmation email shortly.

Sign up and receive emails about and our online training library:

Here’s our privacy policy with more details about how we handle your information.

Keep up with news, tips, and latest courses with emails from

Sign up and receive emails about and our online training library:

Here’s our privacy policy with more details about how we handle your information.

submit Lightbox submit clicked
Terms and conditions of use

We've updated our terms and conditions (now called terms of service).Go
Review and accept our updated terms of service.