Saturday, May 22, 2010

The directory service cannot replicate with this server because the time since the last replication with this server has exceeded the tombstone lifeti

SkyHi @ Saturday, May 22, 2010

Solution

Treat this occurrence as a lingering object condition, and do the following:

  • Run the repadmin /showrepl command on the domain controller that received the error to determine which domain controller has been disconnected for longer than a tombstone lifetime.

  • Remove lingering objects. Follow the instructions for removing lingering objects from the source and destination domain controllers as described in Event ID 1388 or 1988: A lingering object is detected.

  • Restart replication on the destination domain controller. After you remove lingering objects, you must restart replication on the domain controller that logged the event by editing the registry setting that allows replication with a potentially out-of-date domain controller. You can also perform this procedure if you do not want to wait to remove lingering objects and you want to start replication immediately.

  • Reset the registry to protect the domain controller against outdated replication. After replication has resumed on the domain controller that logged the event, reset the registry so that this domain controller continues to log events if replication is attempted with a domain controller where the last successful replication occurred longer than a tombstone lifetime ago.

Restart Replication Following Event ID 2042

To restart inbound replication on the destination domain controller following event ID 2042, you must edit the Allow Replication With Divergent and Corrupt Partner registry entry in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters.

Use the following procedure to change the registry entry value. This procedure does not require a restart of the domain controller to take effect.

REFERENCES
http://technet.microsoft.com/en-us/library/cc757610%28WS.10%29.aspx
====================================================================
====================================================================


C:\Users\Administrator.W2K8>repadmin /showrepl

Repadmin: running command /showrepl against full DC localhost
Vancouver\W2K8AD2
DSA Options: IS_GC
Site Options: (none)
DSA object GUID: c860e2a8-e512-4b11-be91-600bf110c339
DSA invocationID: fc20ac7b-701a-4d1d-995e-c7e4f88106b1

==== INBOUND NEIGHBORS ======================================

DC=w2k8,DC=local
Vancouver\W2K8AD1 via RPC
DSA object GUID: ad9c2f9d-1236-43cd-9c79-cea6eb7d945a
Last attempt @ 2010-05-22 21:48:53 failed, result 8614 (0x21a6):
The directory service cannot replicate with this server because the time since the last replication with this server has exceeded the tombstone lifetime.
169 consecutive failure(s).
Last success @ 2009-09-26 15:02:24.

CN=Configuration,DC=w2k8,DC=local
Vancouver\W2K8AD1 via RPC
DSA object GUID: ad9c2f9d-1236-43cd-9c79-cea6eb7d945a
Last attempt @ 2010-05-22 21:47:10 failed, result 8614 (0x21a6):
The directory service cannot replicate with this server because the time since the last replication with this server has exceeded the tombstone lifetime.
8 consecutive failure(s).
Last success @ 2009-09-26 14:50:50.

CN=Schema,CN=Configuration,DC=w2k8,DC=local
Vancouver\W2K8AD1 via RPC
DSA object GUID: ad9c2f9d-1236-43cd-9c79-cea6eb7d945a
Last attempt @ 2010-05-22 21:47:10 failed, result 8614 (0x21a6):
The directory service cannot replicate with this server because the time since the last replication with this server has exceeded the tombstone lifetime.
8 consecutive failure(s).
Last success @ 2009-09-26 14:45:17.

DC=DomainDnsZones,DC=w2k8,DC=local
Vancouver\W2K8AD1 via RPC
DSA object GUID: ad9c2f9d-1236-43cd-9c79-cea6eb7d945a
Last attempt @ 2010-05-22 21:47:10 failed, result 8614 (0x21a6):
The directory service cannot replicate with this server because the time since the last replication with this server has exceeded the tombstone lifetime.
15 consecutive failure(s).
Last success @ 2009-09-26 14:45:17.

DC=ForestDnsZones,DC=w2k8,DC=local
Vancouver\W2K8AD1 via RPC
DSA object GUID: ad9c2f9d-1236-43cd-9c79-cea6eb7d945a
Last attempt @ 2010-05-22 21:47:10 failed, result 8614 (0x21a6):
The directory service cannot replicate with this server because the time since the last replication with this server has exceeded the tombstone lifetime.
8 consecutive failure(s).
Last success @ 2009-09-26 14:45:17.

Source: Vancouver\W2K8AD1
******* 162 CONSECUTIVE FAILURES since 2009-09-26 15:02:24
Last error: 8614 (0x21a6):
The directory service cannot replicate with this server because the time since the last replication with this server has exceeded the tombstone lifetime.



repadmin /removelingeringobjects
C:\Users\Administrator.W2K8>repadmin /removelingeringobjects
Invalid arguments.

Removes lingering objects - an object stored in Active Directory that has
seen, deleted and garbage collected by a reference DC but continues to
incorrectly exist on direct or transitive replication partners DC's that
have not inbound replicated knowledge of the objects deletion within
tombstone lifetime number of days.

The PC running repadmin may have Windows Vista or Windows Server 2008
installed, and must have network connectivity to all domain controllers
targeted by the parameter.

The reference DC must host a writeable copy of the directory partition
targeted for lingering object removal and have network connectivity to all
domain controllers targeted by the parameter.

DC's targeted by the parameter may host read-only or writeable
copies of directory partition targeted for lingering object removal.

DC's and Global catalogs targeted by continue to advertise
and service ldap request during lingering object removal.

The reference DC and domain controllers targeted by the
parameter may have Windows Server 2003, Windows Server 2003 R2 or
Windows Server 2008 installed.



There are no domain or forest functional requirements for this command.

ADVISORY_MODE is a test mode that logs NTDS Replication events 1938, 1946
and 1942 in the targeted domain controllers' directory service event log
identifying the lingering objects that should be removed but does not
actually remove them.

Lingering objects are removed when "repadmin /removelingeringobjects" is
run without the /advisory_mode switch. NTDS Replication events 1937, 1945
and 1939 logged on the target DC's directory services event log identify
the start, conclusion and set of objects removed from a directory
partition.

You should conceptually think of DC's in the as the "bad" DC's
that you want to test or remove lingering objects from and
as the "reference" DC.

Microsoft recommends enabling strict replication consistency before
removing lingering objects.



[SYNTAX]

/removelingeringobjects [/ADVISORY_MODE]

[EXAMPLES]

The following command would check the Europe NC on all DC's in the site HQ
for lingering objects using the DC specified by its ObjectGUID
667f7037-8198-4357-8f15-8f709f04b6e2 as reference.

The /ADVISORY_MODE will cause events to be written to the Directory Service
Event Log for each of the target DC's indicating how many lingering objects
were found.

/removelingeringobjects site:HQ 667f7037-8198-4357-8f15-8f709f04b6e2 DC=europe,DC=contoso,DC=com /ADVISORY_MODE

The following command would check and remove lingering objects from the
Europe NC on DC dubdc03 using the DC specified by
ObjectGUID 667f7037-8198-4357-8f15-8f709f04b6e2.

/removelingeringobjects dubdc03.contoso.com 667f7037-8198-4357-8f15-8f709f04b6e2 DC=europe,DC=contoso,DC=com



C:\Users\Administrator.W2K8>






C:\Users\Administrator.W2K8>repadmin /removelingeringobjects w2k8ad2.w2k8.local ad9c2f9d-1236-43cd-9c79-cea6eb7d945a DC=w2k8,DC=local
RemoveLingeringObjects successful on w2k8ad2.w2k8.local.

C:\Users\Administrator.W2K8>repadmin /removelingeringobjects w2k8ad2.w2k8.local ad9c2f9d-1236-43cd-9c79-cea6eb7d945a CN=Configuration,DC=w2k8,DC=local
RemoveLingeringObjects successful on w2k8ad2.w2k8.local.

C:\Users\Administrator.W2K8>repadmin /removelingeringobjects w2k8ad2.w2k8.local ad9c2f9d-1236-43cd-9c79-cea6eb7d945a CN=Schema,CN=Configuration,DC=w2k8,DC=local
RemoveLingeringObjects successful on w2k8ad2.w2k8.local.

C:\Users\Administrator.W2K8>repadmin /removelingeringobjects w2k8ad2.w2k8.local ad9c2f9d-1236-43cd-9c79-cea6eb7d945a DC=DomainDnsZones,DC=w2k8,DC=local
RemoveLingeringObjects successful on w2k8ad2.w2k8.local.

C:\Users\Administrator.W2K8>repadmin /removelingeringobjects w2k8ad2.w2k8.local ad9c2f9d-1236-43cd-9c79-cea6eb7d945a DC=ForestDnsZones,DC=w2k8,DC=local
RemoveLingeringObjects successful on w2k8ad2.w2k8.local.

C:\Users\Administrator.W2K8>
To restart replication following event ID 2042

1. Click Start, click Run, type regedit, and then click OK.
2. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters

3. If the registry entry does not exist, create the entry as follows:
1. Right-click Parameters, click New, and then click DWORD Value.
2. Type the name Allow Replication With Divergent and Corrupt Partner, and then press ENTER.
3. Double-click the entry. In the Value data box, type 1, and then click OK.
Reset the Registry to Protect Against Outdated Replication
When you are satisfied that lingering objects have been removed and replication has occurred successfully from the source domain controller, edit the registry to return the value in Allow Replication With Divergent and Corrupt Partner to 0.

4.

5.

Split Name Server DNS Configuration Types

SkyHi @ Saturday, May 22, 2010

Most DNS servers are schizophrenic - they may be masters (authoritative) for some zones, slaves for others and provide caching or forwarding for all others. Many observers object to the concept of DNS types partly because of the schizophrenic behaviour of most DNS servers and partly to avoid confusion with the name.conf zone parameter 'type' which only allows master, slave, stub, forward, hint). Nevertheless, the following terms are commonly used to describe the primary function or requirement of DNS servers.

Notes:

  1. Running any DNS server that does not need to support recursive queries for external users (an Open DNS) is a bad idea. While it may look like a friendly and neighbourly thing to do it carries with it a possible threat that it may be used in DDoS attacks as well as an increased risk of cache poisoning. The various configurations have been modified to ensure that the DNS stays Closed to non-permitted users.

  2. One of the basic rules of security is that only the minimum services necessary to meet the objectives should be deployed. This means that a secure DNS server should provide only a single function, for instance, authoritative only, or caching only, not both capabilities in the same server. This is a correct but idealistic position, generally possible only in larger organizations. In practice many of us run mixed mode DNS servers. While much can be done to mitigate any security implications it must always be accepted that, in mixed configurations, increased risk is the downside of flexibility.

Contents

4.1 Master (Primary) Name Servers

A Master DNS defines one or more zone files for which this DNS is Authoritative ('type master'). The zone has been delegated (via an NS Resource Record) to this DNS.

The term 'master' was introduced in BIND 8.x and replaced the term 'primary'.

Master status is defined in BIND by including 'type master' in the zone declaration section of the named.conf file) as shown by the following fragment.

// example.com fragment from named.conf
// defines this server as a zone master
zone "example.com" in{
type master;
file "pri.example.com";
};

Notes:

  1. The terms Primary and Secondary DNS entries in Windows TCP/IP network properties mean nothing, they may reflect the 'master' and 'slave' name-server or they may not - you decide this based on operational need, not BIND configuration.

  2. It is important to understand that a zone 'master' is a server which gets its zone data from a local source as opposed to a 'slave' which gets its zone data from an external (networked) source (typically the 'master' but not always). This apparently trivial point means that you can have any number of 'master' servers for any zone if it makes operational sense. You have to ensure (by a manual or other process) that the zone files are synchronised but apart from this there is nothing to prevent it.

  3. Just to confuse things still further you may run across the term 'Primary Master' this has a special meaning in the context of dynamic DNS updates and is defined to be the name server that appears in the SOA RR record.

When a master DNS receives Queries for a zone for which it is authoritative then it will respond as 'Authoritative' (AA bit is set in a query response).

If a DNS server receives a query for a zone for which it is neither a Master nor a Slave then it will act as configured (in BIND this behaviour is defined in the named.conf file):

  1. If caching behaviour is permitted and recursive queries are allowed the server will completely answer the request or return an error.
  2. If caching behaviour is permitted and Iterative (non-recursive) queries are allowed the server can respond with the complete answer (if it is already in the cache because of another request), a referral or return an error.
  3. If caching behaviour is NOT permitted (an 'Authoritative Only' DNS server) the server will return a referral or an error.

A master DNS server can NOTIFY zone changes to defined (typically slave) servers - this is the default behaviour. NOTIFY messages ensure zone changes are rapidly propagated to the slaves (interrupt driven) rather than rely on the slave server periodically polling for changes. The BIND default is to notify the servers defined in NS records for the zone - except itself, obviously.

A zone master can be 'hidden' (only one or more of the slaves know of its existence). There is no requirement in such a configuration for the master server to appear in an NS RR for the domain. The only requirement is that two (or more) name servers support the zone. Both servers could be any combination of master-slave, slave-slave or even master-master.

If you are running Stealth Servers and wish them to be notified you will have to add an also-notify parameter as shown in the BIND named.conf file fragment below:

// example.com fragment from named.conf
// defines this server as a zone master
// 192.168.0.2 is a stealth server NOT listed in a NS record
zone "example.com" in{
type master;
also-notify {192.168.0.2;};
file "pri/pri.example.com";
};

You can turn off all NOTIFY operations by specifying 'notify no' in the zone declaration.

Example configuration files for a master DNS are provided.

up icon

4.2 Slave (Secondary) Name Servers

A Slave DNS gets its zone data using a zone transfer operation (typically from a zone master) and it will respond as authoritative for those zones for which it is defined to be a 'slave' and for which it has a currently valid zone configuration. It is impossible to determine from a query result that it came from a zone master or slave.

The term 'slave' was introduced in BIND 8.x and replaced the term 'secondary'.

There can be any number of slave DNS's for any given zone.

Slave status is defined in BIND by including 'type slave' in the zone declaration section of the named.conf file as shown by the following fragment.

// example.com fragment from named.conf
// defines this server as a zone slave
zone "example.com" in{
type slave;
file "sec/sec.example.com";
masters {192.168.23.17;};
};

Notes:

  1. The master DNS for each zone is defined in the 'masters' statement of the zone clause and allows slaves to refresh their zone record when the 'expiry' parameter of the SOA Record is reached. If a slave cannot reach the master DNS when the 'expiry' time has been reached it will stop responding to requests for the zone. It will NOT use time-expired data.
  2. The file parameter is optional and allows the slave to write the transferred zone to disc and hence if BIND is restarted before the 'expiry' time the server will use the saved data. In large DNS systems this can save a considerable amount of network traffic.

Assuming NOTIFY is allowed in the master DNS for the zone (the default behaviour) then zone changes are propagated to all the servers defined with NS Records in the zone file. Other acceptable NOTIFY sources can be defined using the allow-notify parameter in named.conf.

Example configuration files for a slave DNS are provided.

up icon

4.2.1 But Slaves can also be Masters

Oh, stop this pain. This section can get a bit confusing. Read it only when accompanied by your favorite keep-me-awake-cos-I-can't-take-anymore-of-this-stuff beverage.

The definition of a slave server is simply that it gets its zone data via zone transfer whereas a master gets its zone data from a local file system. The source of the zone transfer could just as easily be another slave as a master.

So what sane human would want to do that?

  1. Assume you want to hide your master servers in, say, a stealth configuration then at least one slave server will sit on the public side of a firewall, or similar configuration, providing perimeter defence. To provide resilience you would need two or more such public slaves. The second slave can be updated from the same master as the first or it could be updated from the slave server - we'll call it the 'boss' slave to avoid getting into tortuous terminology (is it a master-slave or a slave-master?). To configure this miracle the second slave server would define the 'boss' slave's IP in its masters statement. When the 'boss' slave has sucessfully transfered a zone file (from the master) it will send out NOTIFY messages (the default) unless configured not to do so. This type of configuration will marginally increase latency for updating the zone on the second slave - but that may be more than offset by increased stealth.

  2. In a DNSSEC environment the master will likely have all kinds of whizzo dodads concerned with keeping keys secure. Whereas DNSSEC slaves simply send the data in the zone file in response to queries and have no requirements for secure key maintenance. Hidden master configurations will become increasingly the norm in this environment.

up icon

4.3 Caching Name Servers

A Caching Server obtains information from another server (a Zone Master) in response to a host query and then saves (caches) the data locally. On a second or subsequent request for the same data the Caching Server will respond with its locally stored data (the cache) until the time-to-live (TTL) value of the response expires at which time the server will refresh the data from the zone master.

If the caching server obtains its data directly from a zone master it will respond as 'authoritative', if the data is supplied from its cache the response is 'non-authoritative'.

The default BIND behaviour is to cache and this is associated with the recursion parameter (the default is 'recursion yes'). There are many configuration examples which show caching behaviour being defined using a type hint statement in a zone declaration. These configurations confuse two distinct but related functions. If a server is going to provide caching services then it must provide recursive queries and recursive queries need access to the root servers which is provided via the 'type hint' statement. A caching server will typically have a named.conf file which includes the following fragment:

// options section fragment of named.conf
// recursion yes is the default and may be omitted
options {
directory "/var/named";
version "not currently available";
recursion yes;
};
// zone section
....
// the DOT indicates the root domain = all domains
zone "." IN {
type hint;
file "root.servers";
};

Notes:

  1. BIND defaults to recursive queries which by definition provides caching behaviour. The named.conf recursion parameter controls this behaviour.
  2. The zone '.' is shorthand for the root domain which translates to 'any domain not defined as either a master or slave in this named.conf file'.
  3. cache data is discarded when BIND is restarted.

The most common DNS server caching configurations are:

  • A DNS server acting as master or slave for one or more zones (domains) and as cache server for all other requests. A general purpose DNS server.
  • A caching only local server - typically used to minimise external access or to compensate for slow external links. This is sometimes called a Proxy server though we prefer to associate the term with a Forwarding server

To cache or not is a crucial question in the world of DNS. BIND is regarded as the reference implementation of the DNS specification. As such it provides excellent - if complex to configure - functionality. The down side of generality is suboptimal performance on any single function - in particular caching involves a non-trivial performance overhead.

For general usage the breadth of BIND functionality typically offsets any performance concerns. However if the DNS is being 'hit' thousands of times per second performance is a major factor. There are now a number of alternate Open Source DNS servers some of which stress performance. These servers typically do NOT provide caching services (they are said to be 'Authoritative only' servers).

Example configuration files for a caching DNS are provided.

Note: The response to a query is Authoritative under three conditions:

  1. The response is received from a Zone master.
  2. The response is received from a Zone slave with non time-expired zone data.
  3. The response is received by a caching server directly from either a Zone master or slave. If the response is read from the cache directly it is not authoritative.

up icon

4.4 Forwarding (a.k.a Proxy) Name Servers

A forwarding (a.k.a. Proxy, Client, Remote) server is one which simply forwards all requests to another DNS and caches the results. On its face this look a pretty pointless exercise. However a forwarding DNS sever can pay-off in two ways where access to an external network is slow or expensive:

  1. Local DNS server caching - reduces external access and both speeds up responses and removes unnecessary traffic.
  2. Remote DNS server provides recursive query support - reduction in traffic across the link - results in a single query across the network.

Forwarding servers also can be used to ease the burden of local administration by providing a single point at which changes to remote name servers may be managed, rather than having to update all hosts.

Forwarding can also be used as part of a Split Server configuration for perimeter defence.

BIND allows configuration of forwarding using the forward and forwarders parameters either at a 'global' level (in an options section) or on a per-zone basis in a zone section of the named.conf file. Both configurations are shown in the examples below:

Global Forwarding - All Requests

// options section fragment of named.conf
// forwarders can have multiple choices
options {
directory "/var/named";
version "not currently available";
forwarders {10.0.0.1; 10.0.0.2;};
forward only;
};
// zone file sections
....

Per Domain Forwarding

// zone section fragment of named.conf
zone "example.com" IN {
type forward;
forwarders {10.0.0.1; 10.0.0.2;};
};

Where dial-up links are used with DNS forwarding servers BIND's general purpose nature and strict standards adherence may not make it an optimal solution. A number of the Alternate DNS solutions specifically target support for such links. BIND provides two parameters dialup and heartbeat-interval (neither of which is currently supported by BIND 9) as well as a number of others which can be used to minimise connection time.

Example configuration files for a forwarding DNS are provided.

up icon

4.5 Stealth (a.k.a. DMZ or Split) Name Server

A stealth server is defined as being a name server which does not appear in any publicly visible NS Records for the domain. The stealth server is normally used in a configuration called Split Severs which can be roughly defined as having the following characteristics:

  1. The organisation needs a public DNS to enable access to its public services e.g. web, mail ftp etc..
  2. The organisation does not want the world to see any of its internal hosts either by interrogation (query or zone transfer) or should the DNS service be compromised.

A Split Server configuration is shown in Figure 4.1.

Split (Stealth) Server configuration

Figure 4.1 Split Server configuration

The external server(s) is(are) configured to provide Authoritative Only responses and no caching (no recursive queries accepted). The zone file for this server would be unique and would contain ONLY those systems or services that are publicly visible e.g. SOA, NS records for the public (not stealth) name servers, MX record(s) for mail servers and www and ftp service A records. Zone transfers can be allowed between between the public servers as required but they MUST NOT transfer or accept transfers from the Stealth server. While this may seem to create more work, the concern is that should the host running the external service be compromised then inspection of the named.conf or zone files must provide no more information than is already publically visible. If 'master', 'allow-notify','allow-transfer' options are present in named.conf (each of which will contain a private IP) then the attacker has gained more knowledge about the organisation - they have penetrated the 'veil of privacy'.

There are a number of articles which suggest that the view statement may be used to provide similar functionality using a single server but this does not address the problem of the DNS host system being compromised and by simple inspection of the named.conf file additional data about the organisation could be discovered. In our opinion 'view' does not provide adequate security in a 'Split DNS' solution.

A minimal public zone file is shown below:

; public zone master file
; provides minimal public visibility of external services
example.com. IN SOA ns.example.com. root.example.com. (
2003080800 ; se = serial number
3h ; ref = refresh
15m ; ret = update retry
3w ; ex = expiry
3h ; min = minimum
)
IN NS ns1.example.com.
IN NS ns2.example.com.
IN MX 10 mail.example.com.
ns1 IN A 192.168.254.1
ns2 IN A 192.168.254.2
mail IN A 192.168.254.3
www IN A 192.168.254.4
ftp IN A 192.168.254.5

The internal server (the Stealth Server) can be configured to make visible internal and external services, provide recursive queries and all manner of other services. This server would use a private zone master file which could look like this:

; private zone master file used by stealth server(s)
; provides public and private services and hosts
example.com. IN SOA ns.example.com. root.example.com. (
2003080800 ; se = serial number
3h ; ref = refresh
15m ; ret = update retry
3w ; ex = expiry
3h ; min = minimum
)
IN NS ns1.example.com.
IN NS ns2.example.com.
IN MX 10 mail.example.com.
; public hosts
ns1 IN A 192.168.254.1
ns2 IN A 192.168.254.2
mail IN A 192.168.254.3
www IN A 192.168.254.4
ftp IN A 192.168.254.5
; private hosts
joe IN A 192.168.254.6
bill IN A 192.168.254.7
fred IN A 192.168.254.8
....
accounting IN A 192.168.254.28
payroll IN A 192.168.254.29

Using BIND 9's view statement can provide different services to internal and external requests can reduce further the Stealth server's visibility e.g. forwarding all DNS internal requests to the external server.

Example configuration files for a stealth DNS are provided.

up icon

4.6 Authoritative Only Server

The term Authoritative Only is normally used to describe two concepts:

  1. The server will deliver Authoritative Responses - it is a zone master or slave for one or more domains.
  2. The server will NOT cache.

There are two configurations in which Authoritative Only servers are typically used:

  1. As the public or external server in a Split (a.k.a. DMZ or Stealth) DNS used to provide perimeter security.
  2. High Performance DNS servers. In this context general purpose DNS servers such as BIND may not provide an ideal solution and there are a number of Open Source Alternatives some of which specialise in high performance Authoritative only solutions.

You cannot completely turn off caching in BIND but you can control it and provide the functionality described above by simply turning off recursion in the 'option' section of named.conf as shown in the example below.

// options section fragment of named.conf
// recursion no = limits caching
options {
directory "/var/named";
version "not currently available";
recursion no;
};
// zone file sections
....

BIND provides three more parameters to control caching ,max-cache-size and max-cache-ttl neither of which will have much effect on performance in this particular case and allow-recursion which uses a list of hosts that are permitted to use recursion (all others are not).

Example configuration files for a authoritative-only DNS are provided.

up icon

4.7 Split Horizon DNS Server

This section was introduced at the suggestion of Maren Leizaola - many thanks for both taking the time and for providing interesting usage examples.

The term Split Horizon is normally used to describe a DNS server that will give different responses (IP addresses) based on the source address, or some other characteristic, of the query. While it has similar configuration properties to the Stealth DNS it can also be used in a varity of unique situations such as:

  1. Geographic Mapping: Assume that, for example, a web service is replicated in a number of locations (for either performance or access latency reasons) then a specific IP address may be returned based on the source address of the query to ensure the shortest possible path from the user to the service. For those familiar with anycast you could consider this as a poor man's anycast service.

  2. Naming Consistency: Assume that you have, say, a corporate in-house LDAP service and that you want to keep certain highly secure data on one server only accessible to certain individuals or organizational sections, which have unique or identifiable IP addresses or address ranges, but for reasons of consistency (scripts, configuration files etc) you want both the secure and insecure LDAP services to be named, say, ldap.example.com.

Other possibilities may strike imaginative readers. The unifying element is that some characteristic of the incoming query will cause the DNS to generate a query-dependent result.

BIND's view clause provides a method that can be used to build such configurations and example files are provided .


REFERENCES

http://www.zytrax.com/books/dns/ch4/

Windows 7 The remote computer requires Network Level Authentication

SkyHi @ Saturday, May 22, 2010

The "The remote computer requires Network Level Authentication, which your computer does not support." error is what you get when you try to connect to computer running Windows Vista with using recently updated Remote Desktop Client.

It looks like Windows XP doesn't support Network Level Authentication even with the new Remote Desktop Client so you will have to turn NLA off in Vista.

Got to System Properties and select "Allow connections from computer running any version of Remote Desktop (less secure). It's less secure, but it works.


Solution:

To enable NLA in XP machines; first install XP SP3, then edit the registry settings on the XP client machine to allow NLA

• Configure Network Level Authentication

1. Click Start, click Run, type regedit, and then press ENTER.
2. In the navigation pane, locate and then click the following registry subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa
3. In the details pane, right-click Security Packages, and then click Modify.
4. In the Value data box, type tspkg. Leave any data that is specific to other SSPs, and then click OK.
5. In the navigation pane, locate and then click the following registry subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders
6. In the details pane, right-click SecurityProviders, and then click Modify.
7. In the Value data box, type credssp.dll. Leave any data that is specific to other SSPs, and then click OK.
8. Exit Registry Editor.
9. Restart the computer.

Now when you run remote desktop you will notice that Network Level Authentication is supported. To check this, right-click the top left hand corner of a remote desktop session and choose, Help > About



http://support.microsoft.com/kb/951608/

Description of the Credential Security Support
Provider (CredSSP) in Windows XP Service Pack 3

CredSSP is a new Security Support Provider (SSP) that is available in Windows XP SP3 by using the Security Support Provider Interface (SSPI). CredSSP enables a program to use client-side SSP to delegate user credentials from the client computer to the target server. (The target server is accessed by using server-side SSP). Windows XP SP3 involves only the client-side SSP implementation. The client-side SSP implementation is currently being used by Remote Desktop Protocol (RDP) 6.1 Terminal Services (TS). However, the client-side SSP implementation can be used by any third-party program that is willing to use the client-side SSP to interact with programs that are running server-side SSP implementations in Windows Vista or in Windows Server 2008.

To download the CredSSP protocol specification, visit the following Microsoft Web site:
http://download.microsoft.com/download/9/5/E/95EF66AF-9026-4BB0-A41D-A4F81802D92C/%5BMS-CSSP%5D.pdf
(http://download.microsoft.com/download/9/5/e/95ef66af-9026-4bb0-a41d-a4f81802d92c/%5bms-cssp%5d.pdf)
Note By default, CredSSP is turned off in Windows XP SP3.

How to turn on CredSSP

Important This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:
322756 

(http://support.microsoft.com/kb/322756/
)
How to back up and restore the registry in Windows
  1. Click Start, click Run, type regedit, and then press ENTER.
  2. In the navigation pane, locate and then click the following registry subkey:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa
  3. In the details pane, right-click Security Packages, and then click Modify.
  4. In the Value data box, type tspkg. Leave any data that is specific to other SSPs, and then click OK.
  5. In the navigation pane, locate and then click the following registry subkey:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders
  6. In the details pane, right-click SecurityProviders, and then click Modify.
  7. In the Value data box, type credssp.dll. Leave any data that is specific to other SSPs, and then click OK.
  8. Exit Registry Editor.
  9. Restart the computer.

Scenarios for using CredSSP

Scenario 1: Programmatically use the SSP

You can now use CredSSP to perform client-side authentication in Windows XP SP3. You can use CredSSP together with authentication APIs to successfully authenticate the server-side counterpart programs that are running in Windows Vista or in Windows Server 2008.

For more information about the AcquireCredentialsHandle (CredSSP) function, visit the following Microsoft Web site:
http://msdn2.microsoft.com/en-us/library/aa965463(VS.85).aspx
(http://msdn2.microsoft.com/en-us/library/aa965463(VS.85).aspx)
For more information about the InitializeSecurityContext (CredSSP) function, visit the following Microsoft Web site:
http://msdn2.microsoft.com/en-us/library/aa965582.aspx
(http://msdn2.microsoft.com/en-us/library/aa965582.aspx)

Scenario 2: Use Terminal Services to connect to Windows Vista or to Windows Server 2008 from Windows XP SP3

  • Use Terminal Services together with the Single Sign-On experience to connect to a Windows Vista-based computer or to a Windows Server 2008-based computer from a Windows XP SP3-based computer by using default (preset) credentials. This feature requires you to modify registry keys that are related to credential delegation.
  • Use Terminal Services to connect from a Windows XP SP3-based computer to a Windows Vista-based computer or to a Windows Server 2008-based computer when Network Level Authentication (NLA) is enforced.
Note You must turn on CredSSP to successfully use Terminal Services to connect to a NLA-enforced Windows Vista-based computer or to a NLA-enforced Windows Server 2008-based computer from a Windows XP SP3-based computer.

CredSSP Group Policy settings

Windows XP SP3 supports CredSSP Group Policy settings that are specific to credentials delegation as it applies in Windows Vista or in Windows Server 2008. However, the CredSSP Group Policy settings are not available as a Group Policy object (GPO) in Windows XP SP3. The CredSSP Group Policy settings can be applied by creating or by modifying registry entries for the required CredSSP Group Policy setting. The registry entries contain the list of server Service Principal Names (SPNs) for which the associated Group Policy setting applies. Additionally, the registry entries contain the serial number of the servers.

For more information about CredSSP Group Policy settings, visit the following Microsoft Web site:
http://msdn2.microsoft.com/en-us/library/bb204773(VS.85).aspx
(http://msdn2.microsoft.com/en-us/library/bb204773(VS.85).aspx)
The following registry keys correspond to Group Policy settings:
  • HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation

    REG_DWORD: AllowDefaultCredentials
    Value data: 1 (enable) 0 (disable)

    REG_DWORD: ConcatenateDefaults_AllowDefault
    Value data: 1 (enable) 0 (disable)

    HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation\AllowDefaultCredentials

    "<serial_no>"="<server SPN>"
  • HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation

    REG_DWORD: AllowDefCredentialsWhenNTLMOnly
    Value data: 1 (enable) 0 (disable)

    REG_DWORD: ConcatenateDefaults_AllowDefNTLMOnly
    Value data: 1 (enable) 0 (disable)

    HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation\AllowDefCredentialsWhenNTLMOnly

    "<serial_no>"="<server SPN>"
  • HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation

    REG_DWORD: AllowFreshCredentials
    Value data: 1 (enable) 0 (disable)

    REG_DWORD: ConcatenateDefaults_AllowFresh
    Value data: 1 (enable) 0 (disable)

    HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation\AllowFreshCredentials

    "<serial_no>"="<server SPN>"
  • HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation

    REG_DWORD: AllowFreshCredentialsWhenNTLMOnly
    Value data: 1 (enable) 0 (disable)

    REG_DWORD: ConcatenateDefaults_AllowFreshNTLMOnly
    Value data: 1 (enable) 0 (disable)

    HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation\AllowFreshCredentialsWhenNTLMOnly

    "<serial_no>"="<server SPN>"
  • HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation

    REG_DWORD: AllowSavedCredentials
    Value data: 1 (enable) 0 (disable)

    REG_DWORD: ConcatenateDefaults_AllowSaved
    Value data: 1 (enable) 0 (disable)

    HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation\AllowSavedCredentials

    "<serial_no>"="<server SPN>"
  • HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation

    REG_DWORD: AllowSavedCredentialsWhenNTLMOnly
    Value data: 1 (enable) 0 (disable)

    REG_DWORD: ConcatenateDefaults_AllowSavedNTLMOnly
    Value data: 1 (enable) 0 (disable)

    HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation\AllowSavedCredentialsWhenNTLMOnly

    "<serial_no>"="<server SPN>"
  • HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation

    REG_DWORD: DenyDefaultCredentials
    Value data: 1 (enable) 0 (disable)

    REG_DWORD: ConcatenateDefaults_DenyDefault
    Value data: 1 (enable) 0 (disable)

    HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation\DenyDefaultCredentials

    "<serial_no>"="<server SPN>"
  • HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation

    REG_DWORD: DenyFreshCredentials
    Value data: 1 (enable) 0 (disable)

    REG_DWORD: ConcatenateDefaults_DenyFresh
    Value data: 1 (enable) 0 (disable)

    HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation\DenyFreshCredentials

    "<serial_no>"="<server SPN>"
  • HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation

    REG_DWORD: DenySavedCredentials
    Value data: 1 (enable) 0 (disable)

    REG_DWORD: ConcatenateDefaults_DenySaved
    Value data: 1 (enable) 0 (disable)

    HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation\DenySavedCredentials

    "<serial_no>"="<server SPN>"
For example, assume that you want to turn on the Single Sign-On experience when you use Terminal Services to connect to a Windows Vista-based computer or to a Windows Server 2008-based computer from a Windows XP SP3-based computer. In this case, you would add the following registry entries on the Windows XP SP3-based computer:

HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation

REG_DWORD: AllowDefaultCredentials
Value data: 00000001

REG_DWORD: ConcatenateDefaults_AllowDefault
Value data: 00000001

HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\CredentialsDelegation\AllowDefaultCredentials

"1"="TERMSRV/*"

APPLIES TO
  • Microsoft Windows XP Service Pack 3

http://support.microsoft.com/kb/951608/


List (and unshare) folders in Windows 7

SkyHi @ Saturday, May 22, 2010

After installing Windows 7, I was browsing around and found that they had removed the little "shared" icon overlay to let users know that a folder is being shared.

I found a way to list the shared folders and noticed that my computer was sharing a bunch of folders that I didn't want shared. Most importantly, ALL my personal and system files were exposed by default. WTF!?

You can disable them by running "Computer Management" through "compmgmt.msc" and selecting "Shared Folders" > "Shares".

There you can find a list of all shared folders in your computer.

Unfortunately, all your drives and system folders are shared by default because the "Server" service is an absolute cunt. Unsharing them will do nothing, because when the service is restarted, the disabled shares will appear again.


REFERENCES

http://twigstechtips.blogspot.com/2009/08/list-and-unshare-folders-in-windows-7.html

Remove dead screens" message when session exists

SkyHi @ Saturday, May 22, 2010
Hi Frederik,

hmmm, are you sure your screen sessions are alive?

here, when type:

$ screen -list

And have more than one screen session like this:

12:39 [EMAIL PROTECTED] ~> screen -list
There are screens on:
5815.pts-1.cronos (Attached)
6150.pts-7.cronos (Detached)
2 Sockets in /home/leslie/.screen.

and type:

[EMAIL PROTECTED] ~> screen -D -R
There are several suitable screens on:
5815.pts-1.cronos (Attached)
6150.pts-7.cronos (Detached)
Type "screen [-d] -r [pid.]tty.host" to resume one of them.
[EMAIL PROTECTED] ~>

or

[EMAIL PROTECTED] ~> screen -x
There are several suitable screens on:
5815.pts-1.cronos (Attached)
6150.pts-7.cronos (Detached)
Type "screen [-d] -r [pid.]tty.host" to resume one of them.


I only receive the message "Remove dead screens with -wipe",

when I have dead sessions that cant be recovered.


I hope this clarify some aspects ;-)


Cheers

LEslie


REFERENCES
http://www.mail-archive.com/screen-users@gnu.org/msg00094.html

Host CPU is incompatible Longmod: VMWARE ESX Vsphere

SkyHi @ Saturday, May 22, 2010

While attempting to install a 64bit OS in ESX Server you receive the following message

Host CPU is incompatible with the virtual machine’s requirements at CPUID level 0x80000001 register ‘edx’.

host bits: 0000:0000:0000:0000:0000:0000:0000:0000

required: xx1x:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx

Mismatch detected for these features:

*Longmode; required when virtual machine is configured for a 64-bit OS.





Cause:

The Virtualization Technology (VT) Extensions have not been enabled in the server BIOS.



Solution:

To install and configure 64bit operating systems in ESX Server requires that Virtualization Technology (VT) Extensions be enabled in the server BIOS settings.

Change the setting on BIOS for Virtualization.

Clean Up the New Ubuntu Grub2 Boot Menu

SkyHi @ Saturday, May 22, 2010

Ubuntu adopted the new version of the Grub boot manager in version 9.10, getting rid of the old problematic menu.lst. Today we look at how to change the boot menu options in Grub2.


Grub2 is a step forward in a lot of ways, and most of the annoying menu.lst issues from the past are gone. Still, if you’re not vigilant with removing old versions of the kernel, the boot list can still end up being longer than it needs to be.


sshot-1


Note: You may have to hold the SHIFT button on your keyboard while booting up to get this menu to show. If only one operating system is installed on your computer, it may load it automatically without displaying this menu.


Remove Old Kernel Entries


The most common clean up task for the boot menu is to remove old kernel versions lying around on your machine.


In our case we want to remove the 2.6.32-21-generic boot menu entries. In the past, this meant opening up /boot/grub/menu.lst…but with Grub2, if we remove the kernel package from our computer, Grub automatically removes those options.


To remove old kernel versions, open up Synaptic Package Manager, found in the System > Administration menu.


When it opens up, type the kernel version that you want to remove in the Quick search text field. The first few numbers should suffice.


sshot-2


For each of the entries associated with the old kernel (e.g. linux-headers-2.6.32-21 and linux-image-2.6.32-21-generic), right-click and choose Mark for Complete Removal.


sshot-3


Click the Apply button in the toolbar and then Apply in the summary window that pops up. Close Synaptic Package Manager.


The next time you boot up your computer, the Grub menu will not contain the entries associated with the removed kernel version.


sshot-4


Remove Any Option by Editing /etc/grub.d


If you need more fine-grained control, or want to remove entries that are not kernel versions, you must change the files located in /etc/grub.d.


sshot-7


/etc/grub.d contains files that hold the menu entries that used to be contained in /boot/grub/menu.lst. If you want to add new boot menu entries, you would create a new file in this folder, making sure to mark it as executable.


If you want to remove boot menu entries, as we do, you would edit files in this folder.


If we wanted to remove all of the memtest86+ entries, we could just make the 20_memtest86+ file non-executable, with the terminal command


sudo chmod –x 20_memtest86+



sshot-10


Followed by the terminal command


sudo update-grub



sshot-11


Note that memtest86+ was not found by update-grub because it will only consider executable files.


However, instead, we’re going to remove the Serial console 115200 entry for memtest86+…


Open a terminal window Applications > Accessories > Terminal. In the terminal window, type in the command:


sudo gedit /etc/grub.d/20_memtest86+



The menu entries are found at the bottom of this file.


sshot-5


Comment out the menu entry for serial console 115200 by adding a “#” to the start of each line.


sshot-6


Save and close this file. In the terminal window you opened, enter in the command


sudo update-grub



Note: If you don’t run update-grub, the boot menu options will not change!


sshot-8


Now, the next time you boot up, that strange entry will be gone, and you’re left with a simple and clean boot menu.


sshot-9


Conclusion


While changing Grub2’s boot menu may seem overly complicated to legacy Grub masters, for normal users, Grub2 means that you won’t have to change the boot menu that often. Fortunately, if you do have to do it, the process is still pretty easy.


For more detailed information about how to change entries in Grub2, this Ubuntu forum thread is a great resource. If you’re using an older version of Ubuntu, check out our article on how to clean up Ubuntu grub boot menu after upgrades.


REFERENCES

http://www.howtogeek.com/howto/17787/clean-up-the-new-ubuntu-grub2-boot-menu/

Wednesday, May 19, 2010

Postfix Sendmail clean mqueue

SkyHi @ Wednesday, May 19, 2010
Most mail servers will retry sending mail up to 5
days and then stop trying. Ideally they should delete the messages that
are not going to be delivered, but often that does not happen. To
delete the messages that are over 5 days old create and run the
following shell script or just run the command on
commandline.



Assumption: your mail q is located at /var/spool/mqueue



#!/bin/bash
#find /var/spool/mqueue -mtime +5 -exec ls -l {} \;

find /var/spool/mqueue -mtime +5 -exec rm -f {} \;

Tuesday, May 18, 2010

How to config ipv6 on solaris 10 and nevada?

SkyHi @ Tuesday, May 18, 2010
To configure ipv6 on an interface, you can use:

ifconfig e1000g0 inet6 plumb up
ifconfig e1000g0 inet6 addif 2000:2::1/64 up

To set up the routing, use something like:

route add -inet6 2000:1::1 2000:2::2


You may want to take a look at the corresponding section in "Solaris
Administration Guide" for details:

http://docs.sun.com/app/docs/doc/819-3000/ipv6-config-tasks-1?a=view

REFERENCE
http://opensolaris.org/jive/thread.jspa?threadID=67173

Installing IPv6 on Windows XP

SkyHi @ Tuesday, May 18, 2010
IPv6 support is still experimental under Windows XP and the stack has to be enabled manually.


To enable the Windows XP IPv6 stack:
  • From the Windows desktop press the “start” button.
  • Click on “Control Panel”.
  • Assuming that the Control Panel is in classic view mode, click on “Network Connections”.
  • Right click on the connection that needs to have the IPv6 stack enabled and go to “Properties”
  • On the properties window click on the “Install…” button.
  • On the “Select Network Component Type” window, select the “Protocol” option and then click on the “Add…” button.
  • On the “Select Network Protocol” window select “Microsoft TCP/IP version 6” and then click the “Ok” button.


The Microsoft IPv6 stack is now enabled for your network connection.


There is no graphical configuration of IPv6 properties/settings. A command line tool used netsh is used to configure IPv6 for interfaces.


To add or delete an IPv6 Address:
  • From a windows command line invoke the netsh tool by typing “netsh” and then pressing the enter key.
  • Next change the context of netsh to interface by typing “interface” and press enter.
  • Change the context of the interface to ipv6 mode by typing “ipv6” and pressing enter.
  • The command to add an address has the form of “add address [interface=]<string> [address=]<IPv6 Adress>”
a. Example: add address interface="Local Area Connection 2" 2001:1945:feed:deef::1


Deletion can be handled in the same manner by using keyword delete instead of keyword add.


Hope this helps.

REFERENCES
http://forums.techarena.in/networking-security/1098260.htm

Unix find and replace text within all files within a directory

SkyHi @ Tuesday, May 18, 2010

To replace all instances of a string in a directory (subdirectories included) do:


Code:


perl -e "s/FIND/REPLACE/g;" -pi.save $(find path/to/DIRECTORY -type f)






The above will make a backup temp file of your original

If you do not want a temp file with the .save extension then do:




Code:


perl -e "s/FIND/REPLACE/g;" -pi $(find path/to/DIRECTORY -type f)






--------------------

Example:

You want to replace all instances of the word "design" with "dezine" in the directory /public_html/company/info



you can execute the command from document root as


Code:


perl -e "s/design/dezine/g;" -pi.save $(find public_html/company/info -type f)






or you can execute the command from public_html/company/ (a directory above) as:


Code:


perl -e "s/design/dezine/g;" -pi.save $(find info -type f)






------------------------------



The above commands will search all files (.gif, .jpg, .htm, .html, .txt) so you might see some error messages "Can't open *.gif", etc)



Simplified



To search just files of type, .htm without a backup file in the current directory only (no subdirectories) you could use:




Code:


perl -pi -e 's/design/dezine/g' *.htm







# *****************************************************************************************
# find_and_replace_in_files.sh
# This script does a recursive, case sensitive directory search and replace of files
# To make a case insensitive search replace, use the -i switch in the grep call
# uses a startdirectory parameter so that you can run it outside of specified directory - else this script will modify itself!
# *****************************************************************************************

!/bin/bash
# **************** Change Variables Here ************
startdirectory="/home/gare/tmp/tmp2"
searchterm="search"
replaceterm="replaceTerm"
# **********************************************************

echo "******************************************"
echo "* Search and Replace in Files Version .1 *"
echo "******************************************"

for file in $(grep -l -R $searchterm $startdirectory)
do
sed -e "s/$searchterm/$replaceterm/ig" $file > /tmp/tempfile.tmp
mv /tmp/tempfile.tmp $file
echo "Modified: " $file
done


echo " *** Yay! All Done! *** "








REFERENCES
http://forums.devshed.com/unix-help-35/unix-find-and-replace-text-within-all-files-within-a-146179.html

Perl one-liner Add first line to all perl file

SkyHi @ Tuesday, May 18, 2010

Example perl one liners for command line use, a summary of important perl command line arguments, and how to convert between 1-liners and full Perl scripts. This page assumes the reader has a reasonable amount of Perl experience. Consult sites like learn.perl.org and Perl Monks to learn more about Perl, or visit the #perl channel on the Freenode IRC network. Also consider the book Minimal Perl by Tim Maher, which covers one liner style Perl in great detail.

The following examples require a Unix shell, such as zsh. Windows systems will need double quotes in place of single quotes.

perl one liners favor quick command line searching and editing. I recommend the practices outlined in Perl Best Practices when developing scripts or applications with Perl. See also Famous Perl One-Liners Explained for additional discussion of Perl one liners.

Learned something? Blog about it!

Perl Argument Overview

Arguments to perl can alter how Perl processes input. For a complete list of these invocation options, see perlrun.

  • -e specifies Perl expressions. More than one can be used, if needed. Other options should not follow this option.
  • $ perl -e 'print "Hello";' -e 'print " World\n"'
    Hello World

    Under Perl 5.10 and higher, the -E option enables various features:

    % perl -E 'say "Hello World"'
    Hello World

  • -p loops over and prints input.
  • -n loops over and does not print input.
  • -l strips newlines on input, and adds them on output. Use this option by default, unless the newlines need special handling, or for efficiency reasons.
  • Use the -ple or -nle option clusters, depending on whether input data should be printed by default or not. The expression 42 does nothing; these examples show the default behavior, and how to enable printing with a -nle.

    $ echo test | perl -ple 42
    test
    $ echo test | perl -nle 42
    $ echo test | perl -nle 'print'
    test

  • -i causes perl to operate on files in-place, and optionally also backs up the files via -i.bak or whatever. I strongly recommend previewing without the -i option before making permanent changes!
  • Never use the -ie '…' invocation, as the -i option reads the e as the backup filename suffix, not the -e as intended. Construct command lines with -e as the last argument before the expression to avoid these sorts of errors:

    # DOS to Unix text convert (example only, dos2unix much faster)
    $ perl -i -pe 's/\r//g' file

    # Legacy MacOS to Unix text convert
    $ perl -i -pe 's/\r/\n/g' file

    # Unix to DOS text convert (unix2dos much faster)
    $ perl -i -pe 's/\n/\r\n/' file

    I shun the backup filename extension to -i, such as -i.bak. Instead, I store data under version control, and thus can revert changes should a sandbox edit go awry. Version control also offers diff support to sanity check the changes made, and commits to log reasons with changes.

  • -a enables auto-split of input into the @F array.
  • Use perl -lane … when processing input into columns: easy to remember (data split into multiple lanes), and handles line breaks via the -l option. Arrays in Perl start at 0, not 1. Also note .. only handles positive ranges (1..2 not 2..1). Use reverse 1..2 to produce a negative trending series.

    $ echo a b c | perl -lane 'print $F[1]'
    b
    $ echo a b c | perl -lane 'print "@F[0..1]"'
    a b
    $ echo a b c | perl -lane 'print "@F[-2,-1]"'
    b c

    Consult perlvar to learn about @F and other special variables.

    Alternatives such as cut or awk may be more efficient for parsing delimited data. Perl functions like getpwent may be better suited to parsing /etc/passwd data.

  • -F specifies the characters to split on with the -a option. Like -i it takes an argument, so should be used apart from other option sets:
  • $ perl -F: -lane 'print $F[0] if !/^#/' /etc/passwd

  • -0 specifies the input record separator. More on this option later.
  • -M lets you load nifty modules such as File::Slurp or IO::All.
  • Life with CPAN covers methods to install perl modules.

  • -d enables debugging mode. For interactive debugging, run something like perl -d -e42, then enter exit when done. I prefer one liners or scripts to any interactive mode.
  • $ perl -d -e42

    Loading DB routines from perl5db.pl version 1.28
    Editor support available.

    Enter h or `h h' for help, or `man perldebug' for more help.

    main::(-e:1): 42
    DB<1> print "Hello World"
    Hello World
    DB<2> exit
    Debugged program terminated. Use q to quit or R to restart,
    use O inhibit_exit to avoid stopping after program termination,
    h q, h R or h O to get additional info.
    DB<3> q

Example One Liners

Experiment with these to practice the expressions; otherwise, make backups in the event an expression runs amok. The examples assume a Unix Bourne compatible shell (such as zsh); other command lines may require altering the quotes around the Perl code (double quotes for Windows), or changes to support C-like shells (csh, tcsh). For more information on shell commands, see my shell tips page.

Current Filename

The special $ARGV variable holds the current filename, or - when data arrives via the standard input filehandle. See perlvar for more information on special variables like $ARGV. As -nle or -ple run code for each line of input, a special block (such as BEGIN or END) or lookup hash must be used if the filename must only be printed once.

$ wc -l /etc/passwd
36 /etc/passwd
$ perl -nle 'END { print $ARGV }' /etc/passwd
/etc/passwd

$ echo test | perl -nle 'print $ARGV'
-
$ (echo test; echo test2) | perl -nle 'print $ARGV'
-
-

$ perl -nle 'print $ARGV if !$seen{$ARGV}++' /etc/passwd /etc/shells
/etc/passwd
/etc/shells

The filename can be used when sending output to a new command that needs the original filename.

  • Include filename in output
  • If looking for data in multiple files, prefix the output with the filename, so the matches can be linked back to the source file. This example searches for unquoted Perl heredoc expressions (<<EOF instead of a more readable <<"END_USAGE"):

    $ perl -nle 'print "$ARGV:$_" if m/<<\s*[A-Z]/' `find . -type f`

    To then edit the matching files with vi, use:

    $ vi `perl -nle 'print $ARGV if m/<<\s*[A-Z]/' `find . -type f``

  • Sendmail Logs
  • The following example matches Sendmail queue strings followed by from=<>, and prints out the filename and queue identifier. The subsequent shell while loop searches for the queue strings in the original file with grep.

    $ perl -nle 'print "$ARGV $1" if /: (\w{14}): from=<>/' /var/log/maillog* \
    | while read filename queueid; do grep $queueid $filename; done

Strip out lines

Use perl -nle 'print if ! …' to say “print, except for the following cases.” Practical uses include omitting lines matching a regular expression, or removing the first line from a file. For more information on regular expressions in Perl, see perlretut and perlreref. Lookup $. in perlvar. Operator precedence may require the use of unless instead of if ! or parenthesized expressions. See perlop for details.

$ (echo a; echo b) | perl -nle 'print if !/b/'
a
$ (echo a; echo b) | perl -nle 'print unless $. == 1'
b

A warning about the special line number variable $. and multiple files: the eof function must be used to reset $. for each new file, as otherwise the line count increases across the files. The following examples demonstrate this behavior by looping over the input file twice; note the use of close ARGV if eof in the second case.

$ cat input
foo
bar
zot
$ perl -nle 'print $.' input input
1
2
3
4
5
6
$ perl -nle 'print $.; close ARGV if eof' input input
1
2
3
1
2
3

Add a line to a file

Appending data to existing files is easy. So is inserting data into arbitrary locations in a file, such as prepending a new first line to a set of files. In the following case, #!/usr/bin/perl will be added as the first line of all *.pl files in the current directory.

$ perl -i -ple 'print q{#!/usr/bin/perl} if $. == 1; close ARGV if eof' *.pl

mod:

$ perl -i -ple "print q{This's rocks!} if $. == 1; close ARGV if eof" *.pl

If a recursive replace is needed, either investigate the use of the modules File::Find or IO::All, or list all the files via a Unix shell command. If filenames contain spaces, use find -print0 and xargs -0 to avoid filenames being misinterpreted by the shell.

$ perl -i -ple 'print q{#!/usr/bin/perl} if $. == 1; close ARGV if eof' \
`find . -type f -name "*.pl"`


$ find . -type f -name "*.pl" -print0 | \
xargs -0 perl -i -ple 'print q{#!/usr/bin/perl} if $. == 1; close ARGV if eof'

The following trick shows how to replace the second line of a file with some text, but only if that line is blank.

$ perl -ple '$_ = "some text" if $. == 2 and m/^$/; close ARGV if eof'

To pipe null delimited data to perl without using xargs -0, supply no argument to the -0 option to perl:

$ find . -type f -print0 | perl -0 -ne 'print "$_\n"'

To alter the last line of a file with an in-place edit, use the eof function as a test:

$ (echo one; echo two) > test
$ perl -i -ple 'tr/a-z/A-Z/ if eof' test
$ cat test
one
TWO

Home on the range

To match or skip blocks of text, use the .. operator. perlop details this operater. This example prints lines, unless blank:

$ cat input
foo



bar
$ perl -ne 'print unless /^$/../^$/' input
foo
bar

The unless statement is equivalent to if not, but is different from if ! due to the associativity and precedence rules covered in perlop. A benefit of this behavior allows the reduction of runs of blank lines to a single blank line.

$ perl -ne 'print if ! /^$/../^$/' input
foo

bar

Line numbers can also be used with the range operator, for instance to remove the first four lines of a file.

$ perl -nle 'print unless 1 .. 4' input
bar

To match a single line with the range operator, use 5..5.

Altering record parsing

Perl uses the -0 option to allow changing the input record separator. Use -00 to operate in paragraph mode, and -0777 to treat the file as a single line. The paragraphs file contains the -0 documentation from perlrun, used in the following example:

$ perl -00 -ne 'print if /special/' paragraphs
The special value 00 will cause Perl to slurp files in paragraph
mode. The value 0777 will cause Perl to slurp files whole because
there is no legal byte with that value.

Parsing the entire input file as a single line can be used to alter the newlines that otherwise require a range operator to deal with, as shown above. By treating an entire file as a single line, a s///g expression can eliminate runs of blank lines:

$ cat input
foo



bar
$ perl -0777 -pe 's/\n+/\n/g' input
foo
bar

Match some data with Backreferences

Use backreferences to extract matching data. If matching a single expression, such as words from the paragraphs file, use a for loop to print them all:

$ perl -nle 'print for m/\b(\S+)\b/g' paragraphs

A while loop must be used when making multiple backreferences: find the matches, then use the $1 and $2 variables to print the results. Another contrived example: find the words on either side of all the the in a file.

$ perl -nle 'print for m/(\S+)\s+the\s+(\S+)/g' paragraphs
specifies
input
digits,
null
follow
digits.
by
null
use
hexadecimal
where
"H"
use
"-x"
$ perl -nle 'while(m/(\S+)\s+the\s+(\S+)/g){print "$1 $2"}' paragraphs
specifies input
digits, null
follow digits.
by null
use hexadecimal
where "H"
use "-x"

Custom Quoting

Shell quoting may cause problems when writing expressions on the command line. On Unix, wrap Perl expressions in single quotes to prevent unwanted shell interpolation. To use a literal single quote inside a single quoted string, the awkward '\'' syntax ends the single quoted string, include a literal quote, then restart the quoted string:

$ perl -le 'print "'\'' is a single quote"'
' is a single quote

Alternative: use an octal code instead; see ascii(1) for a dictionary of ASCII to octal values.

$ perl -le 'print "\047 is a single quote"'
' is a single quote

The od -bc command will display the octal codes for any data passed to it:

$ perl -le 'print "\047"' | od -bc
0000000 047 012
' \n
0000002
$ perl -le 'print chr for 1..250' | od -bc

Perl also allows different quoting operators, see the “Quote and Quote-like Operators” section under perlop for more information on these.

$ perl -le 'print q{single quoted: $$} . qq{ interpolated: $$}'
single quoted: $$ interpolated: 11506

Output to Multiple Files

To split output among multiple files, change where standard output points at based on some test. For example, the following will split a Unix mail file inbox (in mbox format) into multiple files named filename.*, incrementing a number for each message in the mailbox.

$ perl -pe 'BEGIN { $n=1 } open STDOUT, ">$ARGV.$n" and $n++ if /^From /' inbox

Recursive File Mangling

Either the shell or Perl modules can be used to alter files in subdirectories. Perl modules to use include File::Find or File::Find::Rule, among others. Relevant shell commands on Unix include find(1) and xargs(1). Replace the echo in the examples below with the perl command to run on the files. The -print0 argument to find and xargs -0 will work even if files have spaces in their name, unlike the first case.

$ echo `find . -type f`
$ find . -type f -print0 | xargs -0 echo

Fun with @INC

Search under @INC (see perlvar for more information on this array) to find installed Perl modules. This example will search most of @INC for any module names beginning with Config. The shell backticks collapse the directory list into proper arguments for find(1), and the Perl grep excludes non-existent directories and the current directory from the search.

$ find `perl -le 'print for grep {$_ ne q{.}and -d} @INC'` -name "Config*"

Make use of the related %INC hash to find the location of loaded modules on the underlying filesystem:

$ perl -MCPAN -le 'print $INC{"CPAN.pm"}'
/System/Library/Perl/5.8.6/CPAN.pm

Downloading YouTube Videos

Peteris Krumins posted an excellent discussion on downloading YouTube videos with a Perl one-liner.

Converting One Liners

One liners may be used as quick example code, or could be found in someone’s shell history. This may not be the ideal form for commonly used commands. The following section demonstrates how to convert one liners into Perl scripts.

  • Newline handling
  • The -l command line option can easily be ported, simply list it on the shebang line.

    #!/usr/bin/perl -w -l
    use strict;

  • Loop over input (-pe or -ne)
  • Printing loops can be replaced with a while block that prints by default. For a non-printing loop, remove the print statement.

    #!/usr/bin/perl -w -l
    use strict;

    while (<>) {
    # code from -e expressions here

    print;
    } continue {
    close ARGV if eof;
    }

    Special BEGIN or END blocks can be copied in directly, or placed before and after the while loop in the main namespace.

  • In Place Editing
  • To convert the -i in-place option, use the $^I variable (perldoc perlvar), and ensure the files to be processed are in @ARGV before looping over <>.

    # trick to expand globs in input for systems with poor shells (Windows)
    local @ARGV = map glob, @ARGV;

    local $^I = '.orig';

    while (<>) {
    # code here

    print;
    } continue { close ARGV if eof }

    Consider also the File::AtomicWrite module to help with atomic file writes.

END without END

-ple '$c++; END { print $c }' can also be written as -ple '$c++ }{ print $c'.


REFERENCES

http://sial.org/howto/perl/one-liner/