Easy-to-follow video tutorials help you learn software, creative, and business skills.Become a member

Unicode in regular expressions

From: Using Regular Expressions

Video: Unicode in regular expressions

In the last movie we saw how Unicode allows us to represent multibyte characters, which is useful when we need to use characters that are outside of the Roman alphabet. So while Unicode is a great way of representing the characters, it does create complications for regular expressions. First of all, there's the simple fact that words can be spelled in multiple ways. You could have cafe with or without the accent. So if we were searching for only one of those, we might miss the other spellings. And then more importantly, they can be encoded in multiple ways. Cafe can be encoded as either four or five characters.

Unicode in regular expressions

In the last movie we saw how Unicode allows us to represent multibyte characters, which is useful when we need to use characters that are outside of the Roman alphabet. So while Unicode is a great way of representing the characters, it does create complications for regular expressions. First of all, there's the simple fact that words can be spelled in multiple ways. You could have cafe with or without the accent. So if we were searching for only one of those, we might miss the other spellings. And then more importantly, they can be encoded in multiple ways. Cafe can be encoded as either four or five characters.

We need to be able to look for both of those character encodings. And then that creates problems then for wildcard matching. Remember that when we did our wildcard, we set it match any one character. Well, what happens if something is encoded as two characters, then what is our wildcard supposed to do? It also creates some issues for backtracking as well. If we're moving to the string cafe and we get to the e and it's not a match, then does it back up one character or two characters? That's something that we may not think about but that the regular expression engine has to deal with. And then perhaps most importantly is just the fact that Unicode is relatively new.

This was a pretty recent solution to this problem that grew over time. The regular expressions go back to the late 1960s and the birth of UNIX. They've been around a lot longer. So how do we handle these in regular expressions? Well, there's a Unicode indicator, that's the backslash u, that indicates that we're about to work with a Unicode character. So we have \u followed by a four-digit hexadecimal number. So that's 000 through FFFF; those are the possibilities.

So if we were looking for cafe caf\u00E9--that's that character encoding we saw before, just the last four digits of it-- would match cafe with the accent over it, but not cafe without the accent over it. We've correctly specified we're looking for this Unicode character, this exact character encoding, exactly that one--not e in any of its other forms. Now support for this is in Java, JavaScript, .NET, Python, and Ruby.

Perl and PHP support it, but they use the lowercase x instead of the u. So you can do the exact same thing, just a different character; instead of u for Unicode, that use x. It's not supported in older UNIX tools. So in any of those old tools before Unicode came about, they're not going to support it for the most part. Let's try it out. I think it'll make more sense to you. Inside regexpal, let's start by just entering regular expression. Let's just look for a cafe like this and then we're going to type in cafe, cafe, and this time I want the accented e, which is going to be the Option key, e, followed by the e again.

So that it tells it hey, I'm about to do an accent, and then you actually do the accent. And then after that, let's type it one last time, but this time I want to type it as 2 bytes, cafe, and then right after that I want to type the accent. Now this is a little bit tricky, but I'm going to show you how to do it on a Mac. On the Mac, under the Edit menu, they give you Special Characters as an option. If you choose that, it pops up all the possible characters that you could type. Now I've got it set to Code Tables here, so that it shows me the character numbers.

You'll see, if you scroll down till you get to the 0300 row, you'll see that here's that accent. If I just double-click on it--that's Unicode 0301. Double click on it and you'll see that it added that accent over it. Now it looks the same, but it's actually two different spellings. Now you can see the problem that it created for the regexpal engine, for its code coloring. It told me that this is a match even though it's not. Why is it a match? Well, it matches those first four characters, even though it doesn't match the fifth one. You see how that works? So let's try, here now in the regular expression, let's change this, and let's instead say that we're looking for u followed by 00E9.

Now look which one it picked. Now it said, all right, I'm looking for this specific encoding. It didn't match the second one, just that first one. Let's try the other encoding, 0065. Remember, I told you that's just a plain-old e. That's the character encoding for that. If we want the accent over it, we have to do u0301 after it. Remember, 0301 we saw in that special characters--that was the number for what it represented it. Now it matches just this version, not that version. So you can see why it's tricky. You see why this creates problems.

The main thing I wanted you to get out of this was I wanted to introduce the concept to you for you to realize why regular expressions might or might not find some of these characters and give you some tools so you'd be prepared to deal with it. In the next movie we'll talk about how you could match both of these by using wildcards and properties.

Show transcript

This video is part of

Image for Using Regular Expressions
Using Regular Expressions

59 video lessons · 11664 viewers

Kevin Skoglund
Author

 
Expand all | Collapse all
  1. 2m 18s
    1. Welcome
      56s
    2. Using the exercise files
      1m 22s
  2. 19m 55s
    1. What are regular expressions?
      3m 20s
    2. The history of regular expressions
      6m 40s
    3. Regular expression engines
      2m 44s
    4. Installing an engine
      4m 5s
    5. Notation conventions and modes
      3m 6s
  3. 21m 23s
    1. Literal characters
      6m 39s
    2. Metacharacters
      2m 1s
    3. The wildcard metacharacter
      4m 31s
    4. Escaping metacharacters
      4m 53s
    5. Other special characters
      3m 19s
  4. 31m 26s
    1. Defining a character set
      5m 49s
    2. Character ranges
      4m 49s
    3. Negative character sets
      4m 53s
    4. Metacharacters inside character sets
      5m 12s
    5. Shorthand character sets
      6m 30s
    6. POSIX bracket expressions
      4m 13s
  5. 36m 38s
    1. Repetition metacharacters
      7m 17s
    2. Quantified repetition
      6m 59s
    3. Greedy expressions
      6m 27s
    4. Lazy expressions
      6m 46s
    5. Using repetition efficiently
      9m 9s
  6. 20m 24s
    1. Grouping metacharacters
      4m 14s
    2. Alternation metacharacter
      4m 54s
    3. Writing logical and efficient alternations
      7m 33s
    4. Repeating and nesting alternations
      3m 43s
  7. 19m 19s
    1. Start and end anchors
      7m 21s
    2. Line breaks and Multiline mode
      4m 41s
    3. Word boundaries
      7m 17s
  8. 23m 33s
    1. Backreferences
      8m 57s
    2. Backreferences to optional expressions
      3m 51s
    3. Finding and replacing using backreferences
      7m 16s
    4. Non-capturing group expressions
      3m 29s
  9. 32m 31s
    1. Positive lookahead assertions
      6m 39s
    2. Double-testing with lookahead assertions
      7m 16s
    3. Negative lookahead assertions
      6m 10s
    4. Lookbehind assertions
      6m 26s
    5. The power of positions
      6m 0s
  10. 13m 13s
    1. About Unicode
      4m 19s
    2. Unicode in regular expressions
      4m 41s
    3. Unicode wildcards and properties
      4m 13s
  11. 1h 55m
    1. How to use this chapter
      5m 38s
    2. Matching names
      6m 33s
    3. Matching postal codes
      8m 54s
    4. Matching email addresses
      5m 0s
    5. Matching URLs
      8m 1s
    6. Matching decimal numbers and currency
      6m 45s
    7. Matching IP addresses
      7m 10s
    8. Matching dates
      7m 49s
    9. Matching times
      8m 59s
    10. Matching HTML tags
      8m 34s
    11. Matching passwords
      6m 49s
    12. Matching credit card numbers
      9m 36s
    13. Finding words near other words
      6m 38s
    14. Formatting with Search and Replace, pt. 1
      7m 22s
    15. Formatting with Search and Replace, pt. 2
      4m 15s
    16. Formatting with Search and Replace, pt. 3
      7m 10s
  12. 47s
    1. Goodbye
      47s

Start learning today

Get unlimited access to all courses for just $25/month.

Become a member
Sometimes @lynda teaches me how to use a program and sometimes Lynda.com changes my life forever. @JosefShutter
@lynda lynda.com is an absolute life saver when it comes to learning todays software. Definitely recommend it! #higherlearning @Michael_Caraway
@lynda The best thing online! Your database of courses is great! To the mark and very helpful. Thanks! @ru22more
Got to create something yesterday I never thought I could do. #thanks @lynda @Ngventurella
I really do love @lynda as a learning platform. Never stop learning and developing, it’s probably our greatest gift as a species! @soundslikedavid
@lynda just subscribed to lynda.com all I can say its brilliant join now trust me @ButchSamurai
@lynda is an awesome resource. The membership is priceless if you take advantage of it. @diabetic_techie
One of the best decision I made this year. Buy a 1yr subscription to @lynda @cybercaptive
guys lynda.com (@lynda) is the best. So far I’ve learned Java, principles of OO programming, and now learning about MS project @lucasmitchell
Signed back up to @lynda dot com. I’ve missed it!! Proper geeking out right now! #timetolearn #geek @JayGodbold
Share a link to this course

What are exercise files?

Exercise files are the same files the author uses in the course. Save time by downloading the author's files instead of setting up your own files, and learn by following along with the instructor.

Can I take this course without the exercise files?

Yes! If you decide you would like the exercise files later, you can upgrade to a premium account any time.

Become a member Download sample files See plans and pricing

Please wait... please wait ...
Upgrade to get access to exercise files.

Exercise files video

How to use exercise files.

Learn by watching, listening, and doing, Exercise files are the same files the author uses in the course, so you can download them and follow along Premium memberships include access to all exercise files in the library.


Exercise files

Exercise files video

How to use exercise files.

For additional information on downloading and using exercise files, watch our instructional video or read the instructions in the FAQ.

This course includes free exercise files, so you can practice while you watch the course. To access all the exercise files in our library, become a Premium Member.

Are you sure you want to mark all the videos in this course as unwatched?

This will not affect your course history, your reports, or your certificates of completion for this course.


Mark all as unwatched Cancel

Congratulations

You have completed Using Regular Expressions.

Return to your organization's learning portal to continue training, or close this page.


OK
Become a member to add this course to a playlist

Join today and get unlimited access to the entire library of video courses—and create as many playlists as you like.

Get started

Already a member?

Become a member to like this course.

Join today and get unlimited access to the entire library of video courses.

Get started

Already a member?

Exercise files

Learn by watching, listening, and doing! Exercise files are the same files the author uses in the course, so you can download them and follow along. Exercise files are available with all Premium memberships. Learn more

Get started

Already a Premium member?

Exercise files video

How to use exercise files.

Ask a question

Thanks for contacting us.
You’ll hear from our Customer Service team within 24 hours.

Please enter the text shown below:

The classic layout automatically defaults to the latest Flash Player.

To choose a different player, hold the cursor over your name at the top right of any lynda.com page and choose Site preferencesfrom the dropdown menu.

Continue to classic layout Stay on new layout
Exercise files

Access exercise files from a button right under the course name.

Mark videos as unwatched

Remove icons showing you already watched videos if you want to start over.

Control your viewing experience

Make the video wide, narrow, full-screen, or pop the player out of the page into its own window.

Interactive transcripts

Click on text in the transcript to jump to that spot in the video. As the video plays, the relevant spot in the transcript will be highlighted.

Are you sure you want to delete this note?

No

Your file was successfully uploaded.

Thanks for signing up.

We’ll send you a confirmation email shortly.


Sign up and receive emails about lynda.com and our online training library:

Here’s our privacy policy with more details about how we handle your information.

Keep up with news, tips, and latest courses with emails from lynda.com.

Sign up and receive emails about lynda.com and our online training library:

Here’s our privacy policy with more details about how we handle your information.

   
submit Lightbox submit clicked
Terms and conditions of use

We've updated our terms and conditions (now called terms of service).Go
Review and accept our updated terms of service.