Plan for Today
Anti-Gnashing Gashing Tips
How the Kernel Makes a Process:
Virtual Memory
PS2 is Due Sunday

Exam 1 is out after class Tuesday
(Feb 11) due 11:59pm Thursday
(Feb 13) – open resources, most
questions will be taken from notes
1
Today’s OS News
He started his career as a member of the
technology staff at Sun Microsystems. In
1992, he joined Microsoft. He was on his
way to get a master’s degree in business
when the Microsoft job offer came. The
company was building an operating system
that ultimately would be known as Windows
NT, and needed team members who
understood UNIX and 32-bit operating
systems, he says. Nadella wanted to
complete his master’s degree and take the
Microsoft job. He did both.
2
PS1 Stickers!
Muntaser Ahmed
Benjamin Foster

3
Today’s Experiment!
What should we do to make things better?

“Please excuse the paradox, but stop
listening to young white males like
me. Other people deserve a voice.”

4
Honor Policy
Remember the honor policy: don’t abuse
solutions from last semester
They are easy to find, but viewing them at all is abuse.

5
Foreground Processes
main gash thread

run_command(p)

main gash thread

6
Background Processes
main gash thread
new task
spawn(…)
run_command(p)

main gash thread

7
Parsing Commands
Not our main focus
The main tests will be fairly
simple, but you should
definitely be able to handle
ifconfig | grep "flags" | tail

command := program
command := command &
command := command < file
command := command > file
command := command | command
program := valid program name
file := valid pathname

8
Weilin’s Most Evil Test
curl "http://rust-class.org/pages/ps2.html" | sed "s/[^a-zA-Z ]/ /g" | tr "A-Z " "a-zn"| grep "[a-z]" | sort -u

curl "http://rust-class.org/pages/ps2.html"
| sed "s/[^a-zA-Z ]/ /g" | tr "A-Z " "a-zn"
| grep "[a-z]" | sort -u
Rust Sticker* if you can handle this one!
*: while supplies last!
9
Handling Signals
main gash thread
run_command(p)

Process P

10
Handling Signals
gash Process

Child Process

main gash thread
run_command(p)

Process P

Ctrl-C
11
main gash thread
run_command(p)

Process P

Ctrl-C

Kernel

12
main gash thread
run_command(p)

Process P

signal
handler

Ctrl-C

Kernel

SIGINT
13
main gash thread
run_command(p)

Process P

signal
handler

Ctrl-C

Kernel

SIGINT
14
Handling Signals
Jumping around code like this is inherently
unsafe: you will need to use libc functions
and unsafe
use std::io::signal::{Listener, Interrupt};
use std::libc::funcs::posix88::signal;
…
unsafe { signal::kill(fgpid, libc::SIGINT); }
15
How the Kernel Makes a Process

16
From Class 3:

Program

Batch Processing

Computer
Center

Your Program Runs

Output: Invalid Operation
Charge: $174.32
17
Process Abstraction
Provide each program with the illusion that it
owns the whole machine.
The best example of this way to do things
is Linux, which is an operating system,
which is a program that keeps track of
other programs in a computer and gives
each its due in space and time.
Guy Steele, “How to Grow a Language”
18
Memory Isolation
Physical memory

Process 1 should only be able to access Memory Space 1
Process 2 should only be able to access Memory Space 2

Memory
Space 1

Memory
Space 2

Do we need special hardware support do provide memory isolation?
19
Software-Based Memory Isolation
Original Code

…
movq %rax, -8(%rbp)
…

“Sandboxed” Code

Safe
Loader

…
movq -8(%rbp),%rdx
andq %rdx,%rgx
movq %rax, %rdx
…
Assumes %rdx is reserved and %rgx
is protected and holds a mask for
the memory segment
20
SOSP 1993

21
22
Hardware-Based Memory Isolation
Original Code

Running Code

…
movq %rax, -8(%rbp)
…

…
movq %rax, -8(%rbp)
…

Loader

Memory
Space 1

Memory
Space 2
23
Virtual Memory
address in Process P

Virtual Memory Mapping
physical address
owned by Process P

User-level processes cannot access physical memory directly:
all memory addresses created by process P are virtual addresses,
mapped into physical addresses owned by process P
24
Virtual Memory
address in Process P

Virtual Memory Mapping
physical address
owned by Process P

Who controls the virtual memory mapping?
25
Getting Into the Details…

26
SOSP 1967
Procedure Base Register:
segment number of executing
procedure
Argument Pointer
Base Pointer
Linkage Pointer
Stack Pointer
Descriptor Base Register
27
Generating an Address
18 bits

18 bits

218 = 262144
28
Addressing Mode selects:
Argument Pointer
Base Pointer
Linkage Pointer
Stack Pointer

29
Addressing Mode selects:
Argument Pointer
Base Pointer
Linkage Pointer
Stack Pointer

30
What does
the MULTICS
kernel do to
switch
processes?

31
32
1982

“It used to be that programs were easy to
copy and change. But manufacturers began
to lose money as many people made copies
of software and gave them to their friends.
Now, many manufacturers have figured out
how to 'copy-protect' discs. To our mind
this is a disaster: Most people learn
programming by changing programs to fit
their own needs. This capability of
customization is what makes computers so
attractive. New ways of copy protection will
probably be found soon. Until then, a
computer owner may have to put up with
being 'locked out' of his (sic) own machine.”
Popular Mechanics, January 1982
33
Intel 80186

Intel 80286
First x86 Processor with Virtual Memory Support
“Protected Mode”
34
http://en.wikipedia.org/wiki/File:Intel_i80286_arch.svg
35
Five x86-64 Processor Modes
Real Mode: pretend to be an 8086
20-bit direct-access address space
Protected Mode: “native state”
System Management Mode: platform-specific
power management and security (separate
address space)
Compatibility Mode: pretend to be x86-32
IA-32e/64-bit Mode: run applications in 64-bit
address space

“For brevity, the 64-bit sub-

mode is referred to as 64-bit
mode in IA-32 architecture.”
36
Protected State
(can only be modified
by the kernel):
RFLAGS (includes EFLAGS)
Includes I/O Privilege Level

Control Registers
CR0 bit 0: controls if processor
is in protected mode
CR3: page directory base
register

37
Address Translation

Memory

Physical Address

Linear Address

Paging
Unit
Logical Address

Segmentation
Unit

38
Accessing Memory

16 bits to select segment, 16- 32- or 64- bits to select offset
Actual addressable space for user-level process in Unix:
247 bytes = 128TiB
39
Memory

Paging
Unit

Physical Address

Linear Address

Logical Address

Segmentatio
n Unit

Computing the
Linear Address

Segment selection
is inferred from
instruction type
Logical (“General”, “Virtual”) Address
Segment
Selector

Offset

40
Fetching an Instruction
EIP
CS

Instruction Pointer (Offset)
Code Segment

32 bits

16 bits
13 bits

1

Table Index

Ring
Global or
Local Table

Only Kernel can write to Segment Registers

2

41
Segmentation Tables
Global Descriptor Table (GDT)
limit

base address

linear
address
space
0-264 - 1

Segments can overlap!

13 bits – up to 8192 entries
42
Segmentation Tables
Global Descriptor Table (GDT)
Local Descriptor
Table
(per process)
13 bits – up to 8192 entries

How does the processor find the GDT/LDT?
43
The GDT and LDT are just data
structures in memory!
Special registers store their locations

44
from Class 5
45
Logical Address

Linear Address

Segmentation
Unit

`

Memory

Physical Address

Paging
Unit

46
Linear Address

Logical
Address

64
2

Segmentation
Unit

Paging

linear addresses
What would it cost to have 264 bytes of RAM?
47
$10 per 1GB = 230 bytes

$172B

264 bytes = $10 * 234

Apple’s 2013 revenue
US federal spending for 18 days
48
Paging
Memory

Paging
Unit

Physical Address

Linear Address

Logical Address

Segmentation
Unit

We don’t need to store the whole address space in memory!
Most of it is unused: store rarely-used parts on the disk.
49
Memory

Physical Address

Linear Address

50

Image from Wikipedia

Paging
Unit
Overview (Intel 386)
CR3

32-bit linear address
Offset
Dir
Page
10 bits
(1K tables)

Page
Directory

10 bits
12 bits
(1K entries) (4K pages)
Page Entry

Page Table

Physical
Memory
Page + Offset

CR3+Dir

51
Page Table Entries
CR3
Page Entry

Page Table

Page
Directory

20 bits: physical address
12 bits: flags
user/kernel page
write permission
present

Physical
Memory
Page + Offset

52
386 Checkup 32-bit linear address
CR3

Page
Directory

Physical
Memory

Offset
Dir
Page
10 bits
12 bits
10 bits
(1K tables) (1K entries) (4K pages)

20 bits addr / 12 bits flags

Page Table
Page + Offset

How many pages do we need to store the page table?
53
How slow is this???!
Memory

GDTR

Page

Paging
Unit

Physical
Address

Dir

Linear Address

Logical Address

Segmentation Unit

Offset
CR3

Physical
Memory

Global
Descriptor
Table

Page
Directory

20 bits addr / 12 bits flags

Page Table

Page +
Offset

54
Translation Lookaside Buffer (Cache)

Memory

GDTR

Page

Paging
Unit

Physical
Address

Dir

Linear Address

Logical Address

Segmentation Unit

Offset
CR3

Physical
Memory

Global
Descriptor
Table

Page
Directory

20 bits addr / 12 bits flags

Page Table

Page +
Offset

55
My Favorite Statement in the
Linux Kernel Code!

56
arch/x86/include/asm/tlbflush.h

arch/x86/include/asm/special_insns.h
57
Translation Lookaside Buffer (Cache)

Offset

Memory

GDTR

Page

Paging
Unit

Physical
Address

Dir

Linear Address

Logical Address

Segmentation Unit

flush cache!
CR3

Physical
Memory

Global
Descriptor
Table

Page
Directory

20 bits addr / 12 bits flags

Page Table

Page +
Offset

58
Page Fault
CR3
Page Entry

Page Table

Page
Directory

20 bits: physical address
12 bits: flags
user/kernel page
write permission
present

Physical
Memory

59
How expensive is a page fault?

60
How common are page faults?

61
top -o mem -stats pid,command,cpu,mem,mregion,vsize,faults
62
#include <stdio.h>
#include <stdlib.h>

What will this program do?

int main(int argc, char **argv) {
char *s = (char *) malloc (1);
int i = 0;
while (1) {
printf("%d: %xn", i, s[i]);
i += 4;
}
}

63
#include <stdio.h>
#include <stdlib.h>

What will this program do?

int main(int argc, char > ./a.out {
**argv)
char *s = (char *) malloc (1);
0: 0
int i = 0;
4: 0
while (1) {
8: 0
printf("%d: %xn", i, s[i]);
12: 0
i += 4;
…
}
}
1033876: 0

1033880: 0
1033884: 0
Segmentation fault: 11

64
Charge
Examine the memory use of processes running
on your computer
Problem Set 2 is due Sunday, 9 February
Sign-up for your PS2 demo now/soon!

65

Making a Process (Virtualizing Memory)

  • 2.
    Plan for Today Anti-GnashingGashing Tips How the Kernel Makes a Process: Virtual Memory PS2 is Due Sunday Exam 1 is out after class Tuesday (Feb 11) due 11:59pm Thursday (Feb 13) – open resources, most questions will be taken from notes 1
  • 3.
    Today’s OS News Hestarted his career as a member of the technology staff at Sun Microsystems. In 1992, he joined Microsoft. He was on his way to get a master’s degree in business when the Microsoft job offer came. The company was building an operating system that ultimately would be known as Windows NT, and needed team members who understood UNIX and 32-bit operating systems, he says. Nadella wanted to complete his master’s degree and take the Microsoft job. He did both. 2
  • 4.
  • 5.
    Today’s Experiment! What shouldwe do to make things better? “Please excuse the paradox, but stop listening to young white males like me. Other people deserve a voice.” 4
  • 6.
    Honor Policy Remember thehonor policy: don’t abuse solutions from last semester They are easy to find, but viewing them at all is abuse. 5
  • 7.
    Foreground Processes main gashthread run_command(p) main gash thread 6
  • 8.
    Background Processes main gashthread new task spawn(…) run_command(p) main gash thread 7
  • 9.
    Parsing Commands Not ourmain focus The main tests will be fairly simple, but you should definitely be able to handle ifconfig | grep "flags" | tail command := program command := command & command := command < file command := command > file command := command | command program := valid program name file := valid pathname 8
  • 10.
    Weilin’s Most EvilTest curl "http://rust-class.org/pages/ps2.html" | sed "s/[^a-zA-Z ]/ /g" | tr "A-Z " "a-zn"| grep "[a-z]" | sort -u curl "http://rust-class.org/pages/ps2.html" | sed "s/[^a-zA-Z ]/ /g" | tr "A-Z " "a-zn" | grep "[a-z]" | sort -u Rust Sticker* if you can handle this one! *: while supplies last! 9
  • 11.
    Handling Signals main gashthread run_command(p) Process P 10
  • 12.
    Handling Signals gash Process ChildProcess main gash thread run_command(p) Process P Ctrl-C 11
  • 13.
  • 14.
    main gash thread run_command(p) ProcessP signal handler Ctrl-C Kernel SIGINT 13
  • 15.
    main gash thread run_command(p) ProcessP signal handler Ctrl-C Kernel SIGINT 14
  • 16.
    Handling Signals Jumping aroundcode like this is inherently unsafe: you will need to use libc functions and unsafe use std::io::signal::{Listener, Interrupt}; use std::libc::funcs::posix88::signal; … unsafe { signal::kill(fgpid, libc::SIGINT); } 15
  • 17.
    How the KernelMakes a Process 16
  • 18.
    From Class 3: Program BatchProcessing Computer Center Your Program Runs Output: Invalid Operation Charge: $174.32 17
  • 19.
    Process Abstraction Provide eachprogram with the illusion that it owns the whole machine. The best example of this way to do things is Linux, which is an operating system, which is a program that keeps track of other programs in a computer and gives each its due in space and time. Guy Steele, “How to Grow a Language” 18
  • 20.
    Memory Isolation Physical memory Process1 should only be able to access Memory Space 1 Process 2 should only be able to access Memory Space 2 Memory Space 1 Memory Space 2 Do we need special hardware support do provide memory isolation? 19
  • 21.
    Software-Based Memory Isolation OriginalCode … movq %rax, -8(%rbp) … “Sandboxed” Code Safe Loader … movq -8(%rbp),%rdx andq %rdx,%rgx movq %rax, %rdx … Assumes %rdx is reserved and %rgx is protected and holds a mask for the memory segment 20
  • 22.
  • 23.
  • 24.
    Hardware-Based Memory Isolation OriginalCode Running Code … movq %rax, -8(%rbp) … … movq %rax, -8(%rbp) … Loader Memory Space 1 Memory Space 2 23
  • 25.
    Virtual Memory address inProcess P Virtual Memory Mapping physical address owned by Process P User-level processes cannot access physical memory directly: all memory addresses created by process P are virtual addresses, mapped into physical addresses owned by process P 24
  • 26.
    Virtual Memory address inProcess P Virtual Memory Mapping physical address owned by Process P Who controls the virtual memory mapping? 25
  • 27.
    Getting Into theDetails… 26
  • 28.
    SOSP 1967 Procedure BaseRegister: segment number of executing procedure Argument Pointer Base Pointer Linkage Pointer Stack Pointer Descriptor Base Register 27
  • 29.
    Generating an Address 18bits 18 bits 218 = 262144 28
  • 30.
    Addressing Mode selects: ArgumentPointer Base Pointer Linkage Pointer Stack Pointer 29
  • 31.
    Addressing Mode selects: ArgumentPointer Base Pointer Linkage Pointer Stack Pointer 30
  • 32.
    What does the MULTICS kerneldo to switch processes? 31
  • 33.
  • 34.
    1982 “It used tobe that programs were easy to copy and change. But manufacturers began to lose money as many people made copies of software and gave them to their friends. Now, many manufacturers have figured out how to 'copy-protect' discs. To our mind this is a disaster: Most people learn programming by changing programs to fit their own needs. This capability of customization is what makes computers so attractive. New ways of copy protection will probably be found soon. Until then, a computer owner may have to put up with being 'locked out' of his (sic) own machine.” Popular Mechanics, January 1982 33
  • 35.
    Intel 80186 Intel 80286 Firstx86 Processor with Virtual Memory Support “Protected Mode” 34
  • 36.
  • 37.
    Five x86-64 ProcessorModes Real Mode: pretend to be an 8086 20-bit direct-access address space Protected Mode: “native state” System Management Mode: platform-specific power management and security (separate address space) Compatibility Mode: pretend to be x86-32 IA-32e/64-bit Mode: run applications in 64-bit address space “For brevity, the 64-bit sub- mode is referred to as 64-bit mode in IA-32 architecture.” 36
  • 38.
    Protected State (can onlybe modified by the kernel): RFLAGS (includes EFLAGS) Includes I/O Privilege Level Control Registers CR0 bit 0: controls if processor is in protected mode CR3: page directory base register 37
  • 39.
    Address Translation Memory Physical Address LinearAddress Paging Unit Logical Address Segmentation Unit 38
  • 40.
    Accessing Memory 16 bitsto select segment, 16- 32- or 64- bits to select offset Actual addressable space for user-level process in Unix: 247 bytes = 128TiB 39
  • 41.
    Memory Paging Unit Physical Address Linear Address LogicalAddress Segmentatio n Unit Computing the Linear Address Segment selection is inferred from instruction type Logical (“General”, “Virtual”) Address Segment Selector Offset 40
  • 42.
    Fetching an Instruction EIP CS InstructionPointer (Offset) Code Segment 32 bits 16 bits 13 bits 1 Table Index Ring Global or Local Table Only Kernel can write to Segment Registers 2 41
  • 43.
    Segmentation Tables Global DescriptorTable (GDT) limit base address linear address space 0-264 - 1 Segments can overlap! 13 bits – up to 8192 entries 42
  • 44.
    Segmentation Tables Global DescriptorTable (GDT) Local Descriptor Table (per process) 13 bits – up to 8192 entries How does the processor find the GDT/LDT? 43
  • 45.
    The GDT andLDT are just data structures in memory! Special registers store their locations 44
  • 46.
  • 47.
  • 48.
  • 49.
    $10 per 1GB= 230 bytes $172B 264 bytes = $10 * 234 Apple’s 2013 revenue US federal spending for 18 days 48
  • 50.
    Paging Memory Paging Unit Physical Address Linear Address LogicalAddress Segmentation Unit We don’t need to store the whole address space in memory! Most of it is unused: store rarely-used parts on the disk. 49
  • 51.
  • 52.
    Overview (Intel 386) CR3 32-bitlinear address Offset Dir Page 10 bits (1K tables) Page Directory 10 bits 12 bits (1K entries) (4K pages) Page Entry Page Table Physical Memory Page + Offset CR3+Dir 51
  • 53.
    Page Table Entries CR3 PageEntry Page Table Page Directory 20 bits: physical address 12 bits: flags user/kernel page write permission present Physical Memory Page + Offset 52
  • 54.
    386 Checkup 32-bitlinear address CR3 Page Directory Physical Memory Offset Dir Page 10 bits 12 bits 10 bits (1K tables) (1K entries) (4K pages) 20 bits addr / 12 bits flags Page Table Page + Offset How many pages do we need to store the page table? 53
  • 55.
    How slow isthis???! Memory GDTR Page Paging Unit Physical Address Dir Linear Address Logical Address Segmentation Unit Offset CR3 Physical Memory Global Descriptor Table Page Directory 20 bits addr / 12 bits flags Page Table Page + Offset 54
  • 56.
    Translation Lookaside Buffer(Cache) Memory GDTR Page Paging Unit Physical Address Dir Linear Address Logical Address Segmentation Unit Offset CR3 Physical Memory Global Descriptor Table Page Directory 20 bits addr / 12 bits flags Page Table Page + Offset 55
  • 57.
    My Favorite Statementin the Linux Kernel Code! 56
  • 58.
  • 59.
    Translation Lookaside Buffer(Cache) Offset Memory GDTR Page Paging Unit Physical Address Dir Linear Address Logical Address Segmentation Unit flush cache! CR3 Physical Memory Global Descriptor Table Page Directory 20 bits addr / 12 bits flags Page Table Page + Offset 58
  • 60.
    Page Fault CR3 Page Entry PageTable Page Directory 20 bits: physical address 12 bits: flags user/kernel page write permission present Physical Memory 59
  • 61.
    How expensive isa page fault? 60
  • 62.
    How common arepage faults? 61
  • 63.
    top -o mem-stats pid,command,cpu,mem,mregion,vsize,faults 62
  • 64.
    #include <stdio.h> #include <stdlib.h> Whatwill this program do? int main(int argc, char **argv) { char *s = (char *) malloc (1); int i = 0; while (1) { printf("%d: %xn", i, s[i]); i += 4; } } 63
  • 65.
    #include <stdio.h> #include <stdlib.h> Whatwill this program do? int main(int argc, char > ./a.out { **argv) char *s = (char *) malloc (1); 0: 0 int i = 0; 4: 0 while (1) { 8: 0 printf("%d: %xn", i, s[i]); 12: 0 i += 4; … } } 1033876: 0 1033880: 0 1033884: 0 Segmentation fault: 11 64
  • 66.
    Charge Examine the memoryuse of processes running on your computer Problem Set 2 is due Sunday, 9 February Sign-up for your PS2 demo now/soon! 65