PROBLEM:

Customer enabled LDAP which is working as expected. Now he is seeing a

message in the console every 10 minutes:
01/12/2011 14:42:17 LDAP Server: Bind request for Administrator
failed: Invalid credentials specified: failed to authenticate
BusImpact:
Low impact, no feature failures, just a message (above) in the console
that has the customer concerned.
ACTION TAKEN:
Discussed a similar TN with this message: LDAP Server: Warning: Invalid
credentials specified on Bind request, DN is Test User,O=DOMINOLDAP/
That TN suggests that: This is normal behavior.
Shared with Carl that I have seen (from past experience) where certain
criteria is met, and a message is then written to the console. The
message then repeats every 10 mins. This may be what happened. Carl then
told me that he did do something as 'Administrator' 1x, with no plans to
do this again, he feels that this may have spawned the message after
what I shared.
He is going to bounce the server tonight, so I offered debug to set,
just in case we see it again, and he agreed. Composed and sent this
email:
Hi Carl,
Please enter these commands in the server
console before bouncing the server tonight
by doing this from the i5/OS green screen:
WRKDOMSVR <enter>
Option 8 <enter>
set config DEBUG_CONSOLE_MSG=20,LDAP Server: Bind request for
Administrator
<enter>
set config console_log_enabled=1
<enter>
I'll schedule a follow up for you in one week, if I hear from you
sooner (message appeared) let me know and I'll assist with
collecting the 'live' console.log, and we'll go from there ...
Questions please let me know,

ACTION PLAN: Defined, if the error (msg) is seen again, we should have a
call stack proceeding the message within the console.log. May need to
escalate L3 and ask why it's being thrown every 10 minutes after
conditions were met to throw it once.
ACTION PLAN OWNER: Carl
NEXT COMMUNICATION DATE: 01.19.2010
*********************************************************** LotusCRT01**
+KOPANSKI, KEITH M.
-5724E6200 -L328/-------P3S3-11/01/13-14:15 -AL
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/13-14:15 -CG
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/13-14:15 -AT
+KOPANSKI, KEITH M.
-5724E6200 -L328/-------P3S3-11/01/13-14:15 -AL
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/13-14:15 -CT
-CDDR15 PMRUPDATE BDC -5724E6200 -L503/-------P3S3-11/01/13-14:42 -AT
Material received from HTTP server and stored in ECuRep:
Directory: /ecurep/pmr/0/1/01434,499,000/2011-01-13
File: 01434.499.000.console.log
1164021 bytes
mail address: carln@gofrs.com
-CDDR15 PMRUPDATE BDC -5724E6200 -L328/SDOM40-P3S3-11/01/13-14:43 SCG
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/13-14:44 -CR
S5> SERVICE GIVEN= 99 SG/99/
Email from Carl:
Keith,
Bounced the server, same results.
Uploaded log.nsf to ECUREP.
== End of customer email.
Do not see files uploaded to this PMR so not sure where Carl sent above.
Composed and sent this email to Carl:
Good morning Carl,
What I would prefer to have is the live console.log
which will have the conversation of the server starting
where the new debug was set. Presently the IFS has
this log file locked out, to get around the locks please
follow this document:
http://www-01.ibm.com/support/docview.wss?uid=swg21265422
Once you have the console.log in the /tmp directory, you
may then drag/drop and send to me. Please send using
this link:
http://www.ecurep.ibm.com/app/upload
Within the link, be sure to specify the PMR in its entirety,
Ex: 01434,499,000
Upload is for: LOTUS (vertical drop down)
Questions let me know, I can be reached this morning at
1-610-578-2209 after 9:30am EST.
AP: Await requested console.log - re-q for later work 1.14.2010
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM40-P3S3-11/01/13-14:44 SCT

+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM40-P3S3-11/01/13-14:54 SCC
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM40-P3S3-11/01/13-14:54 SAT
S5> SERVICE GIVEN= 99 SG/99/
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/13-14:54 -CT
-CDDR11 PMRUPDATE BDC -5724E6200 -L503/-------P3S3-11/01/13-15:18 -AT
Material received from HTTP server and stored in ECuRep:
Directory: /ecurep/pmr/0/1/01434,499,000/2011-01-13
File: 01434.499.000.notes.ini
6020 bytes
mail address: carln@gofrs.com
-CDDR11 PMRUPDATE BDC -5724E6200 -L328/SDOM40-P3S3-11/01/13-15:18 SCG
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/13-15:20 -CR
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/13-15:20 -AT
S5> SERVICE GIVEN= 99 SG/99/
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM40-P3S3-11/01/13-15:20 SCT
NO CONTACT IS REQUIRED
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM40-P3S3-11/01/13-15:21
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM40-P3S3-11/01/13-15:21
S5> SERVICE GIVEN= 99 SG/99/
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/13-15:21
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/13-15:35
S5> SERVICE GIVEN= 29 SG/29/
=================================================================
Data collection completed, the debug produced this when the error
was thrown:

SCC
SAT
-CT
-CR

01/12/2011 11:02:15.98 [922574:199773:00008-00000935] LDAP CIServ

Listen> Connection Accepted on Port 389 for Session 22620024
01/12/2011 11:02:15.98 [922574:199773:00010-00000938] LDAP>
InitForSearch
01/12/2011 11:02:15.98 [922574:199773:00005-00000938] LDAP>
InitForSearch
01/12/2011 11:02:15.98 [922574:199773:00010-00000937] LDAP> BERGetTag
State
01/12/2011 11:02:15.98 [922574:199773:00010-00000938] LDAP>
BERGetLeadingLengthByte State
01/12/2011 11:02:15.98 [922574:199773:00010-00000937] LDAP> BERGetNext
State
01/12/2011 11:02:15.98 [922574:199773:00005-00000937] LDAP> Bind State
01/12/2011 11:02:15.98 [922574:199773:00005-00000937] LDAP> Return
Result State (Bind operation)
01/12/2011 11:02:15.98 [922574:199773:00005-00000937] LDAP>
StateReturnResult returning resultCode 49 (Invalid credentials)
01/12/2011 11:02:15.98 [922574:199773:00010-00000937] LDAP>
SendBufferFree
01/12/2011 11:02:15.98 [922574:199773:00005-00000937] LDAP>
InitForSearch
01/12/2011 11:02:15.98 [922574:199773:00010-00000937] LDAP> BERGetTag
State
01/12/2011 11:02:15.98 [922574:199773:00010-00000937] LDAP>
BERGetLeadingLengthByte State
01/12/2011 11:02:15.98 [922574:199773:00010-00000939] LDAP> BERGetNext
State
01/12/2011 11:02:15.98 [922574:199773:00005-00000939] LDAP> UnBind State
01/12/2011 11:02:16 LDAP Server: 192.168.1.7 connected
01/12/2011 11:02:16 LDAP Server: Bind request for Administrator
failed: Invalid credentials specified: failed to authenticate
01/12/2011 11:02:16 LDAP Server: 192.168.1.7 disconnected

Looks as if the server is 'failing' to connect to itself. Requested the

notes.ini from Carl. Found 2 peculiar paramters (one was set 2x). Email
to Carl:
Hi Carl,
Within your notes.ini, I do see a few things that
could be causing this:
DisableLDAPOnAdmin=1
LDAPNoAutoStartRepairDIT=1
LDAPNoAutoStartRepairDIT=1
Is there a reason we have these parameters set?
(the top one is to help 'disable' LDAP) Also, the
second one repeats (it's set twice in the .ini).
Lets remove all 3 please, bounce again, then revisit
whether the error is seen. Please let me know when
you can do this so that I know when to schedule
a follow up AP: Re-q for tomorrow pending updates from Carl. 1.14.2010
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/14-14:14 -CT
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/14-14:44 -CR
S5> SERVICE GIVEN= 29 SG/29/
Carl,
Good morning,
Is 'no news good news' or are we still seeing
the error after the last action plan? Please let
me know,

AP: Await update - re-q for 1.17.2010

+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/14-15:16 -CT
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/14-15:31 -CR
S5> SERVICE GIVEN= 99 SG/99/
Carl is seeing the error and will send me the latest console.log.
-CDDR17 PMRUPDATE BDC -5724E6200 -L503/-------P3S3-11/01/14-15:54 -AT
Material received from HTTP server and stored in ECuRep:
Directory: /ecurep/pmr/0/1/01434,499,000/2011-01-14
File: 01434.499.000.console-DUP0001.log
414360 bytes
mail address: carln@gofrs.com
-CDDR17 PMRUPDATE BDC -5724E6200 -L328/SDOM40-P3S3-11/01/14-15:55 SCG
+BLACK, DARLA
-5724E6200 -L328/SDOM40-P3S3-11/01/14-16:03 SCT
NO CONTACT IS REQUIRED
+BLACK, DARLA
-5724E6200 -L328/SDOM40-P3S3-11/01/14-16:04 SCC
S5> SERVICE GIVEN= 99 SG/99/
Sent email notification to call owner new customer data has arrived.
Closing secondary.
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/14-17:24 -CT
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/14-17:40 -CR
S5> SERVICE GIVEN= 99 SG/99/
Not seeing the stack produced after the bounce. L2 collab to determine
whether this is set correctly:

DEBUG_CONSOLE_MSG=20,LDAP Server: Bind request for Administrator

AP: Pending L2 collab.
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/17-14:41 -CT
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/17-15:45 -CR
S5> SERVICE GIVEN= 99 SG/99/
Perhaps unrelated but seeing this when the server starts:
01/13/2011 23:45:10
Server Init

Xsp Initialization error - Error returned from XSP

Email to Carl:
Good day Carl,
When able, could you please run through the
below document / instructions and let me know
the result?
(Sent: Doc 1442079 How to debug if required PASE and J9 OS programs are
installed correctly on IBM i)
The server may run while you do each test.
Also, please run this command and send the
resultant log file:
DSPSFWRSC OUTPUT(*PRINT)
This will be found via iSeries Navigator Work
Management, Output Queues, QPrint.
Questions let me know please,
AP: Wait for response: Re-q 1.18.2010
-CDDR30 PMRUPDATE BDC -5724E6200 -L503/-------P3S3-11/01/18-14:16 -AT
ECuRep Mail Gateway - mail from support (kopanski@us.ibm.com)
/ecurep/pmr/0/1/01434,499,000/mail20110118-151556-Keith_Kopanski
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/18-14:17 -CT
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/18-15:09 -CR
S5> SERVICE GIVEN= 99 SG/99/
Requested data in looks fine - may mention the XSP error in which a new
PMR would be suggested. Penfing further L2 collab on error:
"LDAP Server: Bind request for Administrator failed: Invalid credentials
specified: failed to authenticate"
AP: Defined. 1.18.2010
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/18-15:31 -CT
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/18-15:42 -CR
S5> SERVICE GIVEN= 99 SG/99/
L2 collab w/ Web Server, composed and sent this to Carl:
Carl,
Thank you.
One more round of debug and server bounce and I
believe that we may become closer. Please set
the following, like this:

WRKDOMSVR <enter>
Option 8 <enter>
set config LDAPDebug=7 <enter>
set config debug_threadid=1 <enter>
Bounce server (end fully, start) and once up please
collect and send the live console.log, again. Much
appreciated, questions please let me know.
AP: Pending requested data. 1.19.2011
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/18-20:02 -CT
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/18-20:10 -CR
S5> SERVICE GIVEN= 99 SG/99/
L2 collab w/ Jennifer G. in which it is believed that this message could
be coming from the iSeries.
Tried to call Carl to walk him through this, went to vm. Sent email
about his availability and when I may call.
We will need to:
Open Ops Nav
Right Click on the server, select Properties
For EACH entry, select Details, then ensure that 'Publish user
informatio' is NOT checked.
After above checks, correction, then via OpsNav, Network, Servers,
TCP/IP, when the right pane populates, ensure that IBM Tivoli Directory
Server for LDAP is NOT running (right click Properties, it should give a
pop-up as to whether this is not running, if running, shut off).
AP: Wait to hear back from Carl on Availability to discuss.
1-19-2011
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/19-15:12 -CT
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/19-15:58 -CR
S5> SERVICE GIVEN= 29 SG/29/
===================================
Called Carl.
Walked him through above.
The IBM Tivoli Server was 'off'.
We did however find 'System' beneath Directory Services checked for
"Publish System Information", we then 'un-checked'.
Carl will upload the 'live' console.log in about an hour. I'll check to
see that the error message stopped every twn minutes, or not. After
which, we may either need to bounce/wait and see - and or close the
ticket, results pending.
AP: Defined - Re-q 1.20.2011
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/19-22:43 -CT
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/19-22:53 -CR
S5> SERVICE GIVEN= 29 SG/29/
Keith,
I think we have it, but my notes log is verbose.
I've attached my notes.ini per your request.

Thanks for the help.

My response:
Thanks Carl,
I looked at your notes.ini for debug that you may remove.
From WRKDOMSVR <enter>
Option 13 <enter>
Page down, put a 'd' in front of this one (deletes), save/exit.
And if you're like me, go back in and make sure it's gone.
DEBUG_CONSOLE_MSG=20,LDAP Server: Bind request for Administrator
Then, WRKDOMSVR <enter>
Option 8 <enter>
Issue these two, one at a time:
set config LDAPDebug=0
set config debug_threadid=0
When the server is bounced, all of the debug that you and I
set will be off, disabled ... Pleasure working with you on this.
If you feel that the message has stopped posting, and I have
permission to close, please advise, and have a great day!
AP: Pending customer response + closure. Will also need to TN this
issue as it is not documented. 1.20.2011
+KOPANSKI, KEITH M.
-5724E6200 -L328/-------P3S3-11/01/19-23:08 -AT
Permission to close - (Note to self) write TN before closing.
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/20-13:59 -CT
NO CONTACT IS REQUIRED
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/20-14:17
S5> SERVICE GIVEN= 99 SG/99/
Defer, later today 1.20.2011
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/20-16:40
+KOPANSKI, KEITH M.
-5724E6200 -L328/-------P3S3-11/01/20-16:45
+KOPANSKI, KEITH M.
-5724E6200 -L328/-------P3S3-11/01/20-16:45
Documented above with new DCF submitted for publishing.

-CR
-CT
-AL
-AT

cc wcc
**********************WEB/eMAIL PROBLEM CLOSURE TEMPLATE****************
Advised customer that problem has been closed as they agreed/requested?
AGREED/REQUESTED:
Requested
Advised customer that if there is anything else I can do for them
regarding this problem or if they have any other issue with this product
or our service that they can call me.
ADVISED?:
YES
Thanked customer for using IBM!YES
No-Survey:
N/A

****************************************************CloseWebPMR*******
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/20-16:46 -CC
+KOPANSKI, KEITH M.
-5724E6200 -L328/SDOM4W-P3S3-11/01/20-16:46 -AT
S5> SERVICE GIVEN= 29 SG/29/