RMAN error – Use CROSSCHECK command to fix status

Our RMAN backup control mail gave an error: x objects could not be deleted for DISK channel(s) due to mismatched status. Use CROSSCHECK command to fix status.

CROSSCHECK is a check to determine whether files on disk or in the media management catalog correspond to the data in the RMAN repository. Because the media manager can mark tapes as expired or unusable, and because files can be deleted from disk or otherwise become corrupted, the RMAN repository can contain outdated information about backups. Crosschecks update outdated RMAN repository information about backups whose repository records do not match their physical status. For example, if a user removes archived logs from disk with an operating system command, the repository still indicates that the logs are on disk, when in fact they are not. The crosscheck command is used to validate RMAN records in the database control file and the recovery catalog against what is physically on the backup media. The crosscheck command can be used on both disk backups and tape backups. You can cross-check the gambit of backups, from database backups and archive-log backups to image copies, the crosscheck command covers them all.

When you run the crosscheck command, any missing backup files will be marked as EXPIRED, meaning that they are no longer on the media where they are expected to be. The list expired command will show you the backups that are expired. You can review this list and then use the delete command to mark the backup files as deleted in the control file and the recovery catalog. The CROSSCHECK command does not delete any files that it is unable to find, but updates their repository records to EXPIRED. Then, you can run DELETE EXPIRED to remove the repository records for all expired files as well as any existing physical files whose records show the status EXPIRED. Expired backups will not show up on this report until the crosscheck command detects they are missing.

Oracle SID should be correctly set, so we'll log on directly.

F:\oracle\rman>rman target =/

We use show all to get the backup retention policy to determine how long backups and archived logs need to be retained for media recovery. You can define a retention policy in terms of backup redundancy or a recovery window. RMAN retains the datafile backups required to satisfy the current retention policy, and any archived redo logs required for complete recovery of those datafile backups. In our case I got RETENTION POLICY TO REDUNDANCY 1 and ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1.

RMAN> show all;

RMAN> delete noprompt obsolete;

Will give the same errors as in our control mail, so let’s do what it wants.

    RMAN> crosscheck archivelog all;

RMAN is not removing all of the files because some of them may still be needed for a full recovery!  It's all about how RMAN defines an obsolete file.  Generally, an obsolete file is one that supplements a full backup that will never be used for a recovery and roll-forward. The Oracle docs note the rules for a file becoming obsolete:
DELETE OBSOLETE does not delete backups required to satisfy the specified retention policy, even if some backups have KEEP UNTIL times set which have passed to override the retention policy.
Backups are never obsolete if they are still needed to meet the retention policy, regardless of any KEEP UNTIL time. With a recovery window-based retention policy, even if the specified KEEP UNTIL time has expired, the backup is retained if the backup is needed to satisfy the recovery window.

With a redundancy-based retention policy, even if the specified KEEP UNTIL time has expired, the backup is retained as long as it is required to satisfy the redundancy requirement. 
You can also use the REDUNDANCY or RECOVERY WINDOW clauses with DELETE to delete backups obsolete under a specific retention policy instead of the configured default:
DELETE OBSOLETE REDUNDANCY = 3;
DELETE OBSOLETE RECOVERY WINDOW OF 7 DAYS;

RMAN> delete noprompt obsolete;

I got a similar error as in the controlemail, this time for the controlefilecopy (32 is here the key):

RMAN> crosscheck controlfilecopy 32;

Now we can delete them:

RMAN> delete noprompt obsolete;

And rerun the last delete to verify if anything’s left:

References:
http://download.oracle.com/docs/cd/B19306_01/backup.102/b14192/maint002.htm
http://users.telenet.be/oraguy.be/rman1.htm
http://download.oracle.com/docs/cd/E11882_01/backup.112/e10643/toc.htm

Lines on Cisco routers

Lines on Cisco routers are physical async serial ports on the router (such as a terminal or modem), a virtual network connection, or another type of serial line on the router. To see which lines you have on your router, use the show line command. Example: show line serial 0/0

Use the show line summary command to get a nice overview.

The console port doesn't need any introduction. The CTY port is, of course, where you configure the router when it’s brand-new — before it has any IP address configuration. The console port is a serial port, so you must have a PC/laptop with a serial interface and connect to the console with a rolled cable, it should be included with your newly arrived devices :)

Once you’ve used the console port to configure the router’s network configuration, it isn’t common to have to use it again. However, it’s good to know that it’s there if anything ever goes wrong. In addition, you should secure the console port to keep someone from connecting to it when you aren’t around.

While not all routers these days have an AUX port, the AUX port is the auxiliary. Think of it as a secondary console port. The AUX ports don’t get a lot of use except to access the router if locked out of the console port. In the past, network admins would connect modems to the AUX ports so they could dial into their routers. Like the console port, the AUX port is a serial port, and you should also take steps to secure it.

VTY ports are virtual TTY ports, used to Telnet or SSH into the router over the network. You can use them to connect to the router to make configuration changes or check the status. Most routers have five VTY ports, numbered 0 to 4. That means you can have up to five concurrent network admins configuring the router at one time.
 

Remember that you can always use the clear line command to clear out a connection on a router line if you run into a problem.

Day 16: Private Networks and NAT

A device directly connected to the Internet has a public IP address, this is routable. The number of public IP addresses is limited so RFC 1918 reserves class A, B and C networks for private use on an internal network. These addresses can be reused for multiple internal networks because the networks are not visible to the Internet or each other.

I've mentioned them a few times but once again a quick overview:

• one class A address for private networks: 10.0.0.0, >16 million private addresses
• 16 class B networks for private use: 172.16.0.0 to 172.31.0.0, each networks allows > 65000 private addresses
• 256 class C networks for private use: 192.168.0.0 to 192.168.255.0, each network allows up to 254 private addresses

A router running NAT and PAT can allow devices on a private network to share a single public IP address and communicate over the Internet. Devices on a private network behind a router running NAT are not directly accessible on the Internet, providing additional security.

Oracle DBA - managing listener.log

As the listener.log file grows, we'll want to remove or rename it. This will fail as it's "being used by another process".
The DBA can simply stop the service, rename/remove the file and restart the service. This can be problematic for users attempting to connect while the listener's down.

Here's a good way to do this without stopping the TNS listener process. This'll work on Windows and with some small changes on Linux as well (ren = mv):
C:\cd \oracle\product\10.2.0\db_1\NETWORK\log
C:\oracle\product\10.2.0\db_1\NETWORK\log\lsnrctl set log_status off
C:\oracle\product\10.2.0\db_1\NETWORK\log\ren listener.log listener.old
C:\oracle\product\10.2.0\db_1\NETWORK\log\lsnrctl set log_status on

In Oracle 11g, the listener log files by default are located in /diag/tnslsnr/product_name/listener. The nice feature about listener log file in this version is, whenever the size of log file grow to 10MB, Oracle starts to writes to a new file. So the log file will not be too large to open for troubleshooting. Overtime, you will have a lot of 10MB log file in the directory. An Oracle DBA needs to manage the listener log files regularly so the log files will not take too much space on the server.

This is a great link with DBA Tips: http://www.idevelopment.info/

Day 17: DNS Operation

All hosts have a HOSTS file that matches names to IP addresses. This file is first used to resolve a request for a domain. Obviously its impossible to keep it all in file, so a DNS server performs this task on a network. If you want to verify the capability to access a DNS server you'd use the nslookup utility: http://technet.microsoft.com/en-us/library/cc725991(WS.10).aspx

The domain naming system has these components: resource records and domain namespace, domain name servers and resolvers. The hierarchy of the domain name system begins at the top with top-level domains such as .com, .org - they could also represent countries. These are followed by second-level domains such as cisco, yet they could also have specific locations such as mail.

So, what happens if a host wants to resolve a Domain Name System (DNS - port 53) name such as mail.cisco.com? A domain that that points to a specific computer in a domain is considered a fully qualified domain name (FQDN). The host uses a resolver to query a DNS server inside its domain to get the IP address of mail.cisco.com. This DNS server is preconfigured (see ipconfig /all). You could use the well known 4.2.2.2 but there's always a security risk using one that you're not authorized to.

1. The DNS server (usually from our ISP) receives the request and checks its local records. If the DNS server cannot resolve the domain name, it forwards the request to another preconfigured DNS server. The local DNS server may query a root DNS server to discover the location of top-level.com domain name servers.
2. The top-level DNS server, after it's queried responds with the location of the cisco.com DNS server for the requested domain.
3. The local DNS server queries the cisco.com DNS server for the location of mail.cisco.com. When the resolved name to IP address is returned, each DNS server caches the record for a limited amount of time.
4. The local DNS server receives the returned request, temporarily caches the record and responds to the requesting host with the IP address for mail.cisco.com.

 It's important to know that DNS zones can be broken into primary or secondary forward lookup or reverse lookup zones:

• Primary and secondary: there can be primary and secondary forward lookup and reverse lookup zones. The primary zone is where you update records, and the secondary zone operates as a read-only backup copy of the primary zone.
• Forward lookup zones: standard zone that resolves FQDNs to IP addresses.
• Reverse lookup zones: this query works in reverse: a host wants to know the FQDN because it knows the IP address. Private networks use reverse lookup to identify host names on their local network. I've had some issues with this lately when we forgot to set this zone up during a rollout of hosts, we got the error Nonexistent domain when testing with nslookup.

It's a good idea to use redundant DNS servers!

There's good information and links on wikipedia: http://en.wikipedia.org/wiki/Domain_Name_System

Day 18: NAT

If you've ever wondered why we're not all using IPv6 yet, the simple answer is Network Address Translation - NAT (NAT overload or PAT).

As seen on day 20, the RPC1918 has identified the private networks. These addresses don't allow us to connect to the Internet, routers cannot route these private IP addresses (they get dropped). A router can however receive a public Internet-routable address from the ISP and provide Internet connectivity for the hosts on the local private network. The router will use NAT to exchange private IP addresses for a public IP address or a pool of public IP addresses. This translation allows an internal host to appear as though it has a public IP address. Nat also provides some basic security.

NAT was developed because of too few available IP addresses. Here are some terms:

• inside local network: the privately addressed internal network connected to a router
• inside local address: internal IP address assigned to a host on the inside, private network. This is usually a private IP address
• outside global network: any network outside the local network that would also not recognize the private addresses assigned to hosts in the local network
• inside global address: a registered, Internet-routable IP address that represents one or more inside local IP addresses to the outside world
• outside local address: destination address of the packet while on the inside local network - typically the same as the outside global address. So it's the IP address of an outside host as it appears to the inside, private network.
• outside global address: actual destination address of the intended external host on the Internet. The IP address assigned to a host on the outside network by the host's owner - usually a routable IP address.

Static NAT translates one private address to one public address. In dynamic NAT will have a pool of public addresses to temporarily assign for internal hosts (the public address will afterwards return to the pool).
A router uses NAT overload, or port address translation (PAT) to allow multiple internal hosts to communicate with just one public IP address. The router uses source port numbers to identify the internal connection request. Note that this implicates that internal hosts must initiate communications with outside networks. If you're wondering what would happen if two hosts used the same source port (from the 65535), it's simple: whoever gets there first gets that port, the other one will get a different one - for example 61751 would then become 61752.

To configure static NAT, you have to designate an inside interface (this is the interface connected to the private network). You also have to designate the interface connected to the outside world as the outside interface. Needed commands:
ip nat outside
ip nat inside
ip nat inside source static local-IP-address global-IP-address

We don't have to know the CLI for ICND1 but I'll give a quick overview, we're preparing for the CCNA - please try it with SDM (use GNS3 if needed).

Router#enable
Router#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
Router(config)#interface serial 1/0
Router(config-if)#ip nat outside
Router(config-if)#interface fa 0/0
Router(config-if)#ip nat inside
Router(config-if)#exit
Router(config)#ip nat inside source static 192.168.1.2 200.1.1.1

NAT with overload enables PAT (a many-to-one mapping). Use the access-list command to define the private address pool that you want to translate to a single IP address. An access list uses a wildcard mask instead of a subnet mask to identify the bits available for use as hosts in the pool. Needed commands (see the differences above):
access-list access-list-number permit  inside-network wildcard-mask
ip nat inside source list access-list-number interface interface overload
ip nat outside
ip nat inside
 
Router(config)#access-list 1 permit 192.168.1.0 0.0.0.255
Router(config)#ip nat inside source list 1 interface serial 1/0 overloadRouter(config)#interface serial 1/0
Router(config-if)#ip nat outside
Router(config-if)#interface fa 0/0
Router(config-if)#ip nat inside
 
Look at 0.0.0.255 as a reversed subnet mask. The overload keyword allows to map multiple IP addresses to a single registered IP address (many-to-one) by using different source port numbers. Use show run and show ip nat translations to verify NAT configuration. A useful command for testing/troubleshooting is debug ip icmp which outputs any ICMP traffic processed by the router. Don't forget to turn it off with undebug all or u all. The clear ip nat translation * command clears all the NAT translations in the NAT table, its useful for troubleshooting.
 
A useful link is http://www.9tut.com/ccna-lab-sim/52-ccna-nat-sim-question

Day 19: Assign Addresses

After having developed a logical topology and proper addressing scheme, we must assign the addresses to devices in our network. A host needs an IP address, default gateway and subnet mask. These can be assigned manually or obtained automatically by a Dynamic Host Configuration Protocol (DHCP) server.

In Windows we'll verify the IP configuration by using ipconfig /all
In Linux use the ifconfig command.

The default gateway for a host is typically the IP address of the connected interface on the router for the network. Each host on the network can then use the router as the gateway to other networks.

This how we assign an IP to the interface of a router (we've already showed it for a switch):

Router>enable
Router#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
Router(config)#interface fastEthernet 0/0
Router(config-if)#ip address 10.0.0.1 255.255.255.0
Router(config-if)#no shutdown
 
The configuration can be verified by using the known commands:
show interfaces
an example is Router#show interfaces fastEthernet 0/0
show running-configuration
show startup-configuration