Escolar Documentos
Profissional Documentos
Cultura Documentos
`whoami`
`whoami`
Francois Loiseau
VCP5 SysAdmin R&D & Cloud
OVH
Big VMware infrastructures High automation level
VMware Users
Linux / Windows / Solaris / Web team Dev team Customers Sysadmins
VMware Administrator
Linux / Windows VMware world Storage: Storage Team Network: Network Team Security: Security Team (Quality) VMware administrators to serve the company
VMware Infrastructures
Self provisionned
Internal cloud
Customer provisonned
vSphere As A Service privateCloud hosting
VMware APIs
vSphere
Perl / Java / powerCli
vCloud
REST
Several ways to do it
To:
/usr/local/lib/perl/5.10.0/
Connexion sample
#!/usr/bin/perl use strict; use Data::Dumper; use VMware::VIRuntime; # Auth options Opts::set_option('server', myvCenter'); Opts::set_option('username', myUser'); Opts::set_option('password', myPassw0rd!');
print "Connecting \n"; Util::connect(); print "Connected \n"; # # [ Well make actions there ] #
9 AM
Quick diagnosis
HA Failure
VM powered on 2 hosts
Quick diagnosis
Vlan segmentation / CMDB
vcenter201
10AM
Storage maintenance planed
Storage team question : May I shut this LUN tomorrow ? I need to rack this SAN somewhere else.
Quick diagnosis
Storage definition / CMDB
vcenter197 vcenter134 vcenter026
{
print $VMView->{'summary.config.name'} . is using . $Device->deviceInfo->summary . \n;
} }
}
}
Win
\o/
2 PM
Security team email :
Dear administrators, We are suspecting unauthorized access to our infrastructures last 3 months. Could you please send us all authentifications / login on VMware front based on this template: Date ; User ; IP From.
: "X"; : "X";
: "X";
On Syslog
Fri Sep 12 11:22:42 2012|vcenter012|Fri Sep 12 11:22:42 2012|User John logged out|X|John|X Fri Sep 12 11:23:52 2012|vcenter012|Fri Sep 12 11:23:52 2012|User John@192.168.0.6 logged in|X|John|X Fri Sep 12 11:23:52 2012|vcenter012|Fri Sep 12 11:23:52 2012|User John logged out|X|John|X Fri Sep 12 11:25:33 2012|vcenter012|Fri Sep 12 11:25:33 2012|User Corp\vscope@172.16.2.2 logged in|X|vscope|X Fri Sep 12 11:25:59 2012|vcenter012|Fri Sep 12 11:25:59 2012|User Franck@46.105.137.3 logged in|X|Franck|X Fri Sep 12 11:26:02 2012|vcenter012|Fri Sep 12 11:26:02 2012|User John@172.16.2.2 logged in|X|John|X Fri Sep 12 11:26:02 2012|vcenter012|Fri Sep 12 11:26:02 2012|User John logged out|X|John|X
? ? ? ? ? ? ? ? ?
: : : : : : : : :
} }
On Syslog
Fri Sep 12 06:51:05 2012|vcenter012|Fri Sep 12 06:51:05 2012|Fri Sep 12 06:51:06 2012|task-13515|Centres de donnes|X|com.vmware.vcIntegrity.CheckNotificationTask|X|success Fri Sep 12 07:01:19 2012|vcenter012|Fri Sep 12 07:01:19 2012|Fri Sep 12 07:01:28 2012|task-13516|nas001223|RefreshDatastoreStorageInfo|Datastore.refreshStorageInfo|Franck|success Fri Sep 12 07:10:55 2012|vcenter012|Fri Sep 12 07:10:55 2012|Fri Sep 12 07:10:57 2012|task-13517|nas000268|RefreshDatastoreStorageInfo|Datastore.refreshStorageInfo|Franck|success Fri Sep 12 11:26:16 2012|vcenter012|Fri Sep 12 11:26:16 2012|Fri Sep 12 11:26:16 2012|task-13525|DatacenterRBX|PowerOnMultiVM_Task|Datacenter.powerOnVm|John|success Fri Sep 12 11:26:24 2012|vcenter012|Fri Sep 12 11:26:24 2012|Task: Reconfigure virtual machine|task-13527|John|queued Fri Sep 12 11:26:24 2012|vcenter012|Fri Sep 12 11:26:24 2012|X|task-13527|NFS99|ReconfigVM_Task|VirtualMachine.reconfigure|John|running Fri Sep 12 11:26:39 2012|vcenter012|Fri Sep 12 11:26:39 2012|X|task-13528|Oracle121b|PowerOffVM_Task|VirtualMachine.powerOff|John|running Fri Sep 12 11:26:59 2012|vcenter012|Fri Sep 12 11:26:59 2012|X|task-13529|Oracle121b|Destroy_Task|VirtualMachine.destroy|John|running
Win
\o/
4 PM
Web team post-production challenges
All VMs with snapshots
Quick diagnosis
Storage definitions / CMDB vCenter definitions
vcenter201 vcenter035 vcenter138
My tought
Get on all web team Filers vmdk & date
find /directory-* |grep .vmdk|grep 0000 |nawk '{FS="\n" ; print $1}' |xargs -i ls -al {}
my $VMViews = Vim::find_entity_views( 'view_type' => 'VirtualMachine', 'begin_entity' => $DatacenterView , ); my $listVm; my $snapshots; my $ret; foreach my $VMView (@$VMViews) { # Checking if VM got at least 1 snapshot if ($VMView->snapshot) { foreach my $snap (@{$VMView->snapshot->rootSnapshotList}) { $ret = _getAllSnapshots( 'snap' => $snap , 'snapshots' => $snapshots ); $listVm->{$VMView->name} = $ret; } } }
# Recursive Sub to list all snaps sub _getAllSnapshots { my %params = @_; my $snap = $params{'snap'}; my $snapshots = $params{'snapshots'}; my my my my ($date1,$date2)= split /T/, $snap->createTime; ($year,$mon,$day) = split /-/, $date1; ($hrs,$min,$sec ) = split /:/, $date2; $createdTime = timegm($sec, $min, $hrs, $day, $mon-1, $year-1900);
# filling result my $snapHash = { 'name' => $snap->name , 'date' => $createdTime, 'desc' => $snap->description }; push @$snapshots , $snapHash; # and go on if ($snap->childSnapshotList) { foreach my $snapChild (@{$snap->childSnapshotList}) {
my $ret = _getAllSnapshots('snap' => $snapChild , 'snapshots' => $snapshots);
}
} return $snapshots; }
Win
\o/
5 PM
Customer request
Notify me on whats going on my infrastructure
Solutions
Send email ? Syslog ?
Let do this
my $msg = User John created on this vCenter. Access has been granted to this Datacenter. We are going to log everything; my $EventManager = Vim::get_view( 'mo_ref' => Vim::get_service_content()->eventManager ); $EventManager->LogUserEvent( 'entity' => $DatacenterView, 'msg' => $msg );
Win
\o/
11 PM
Host Failure
Host 192.168.2.51 pings Failed to ping HARDWARE DRS TRIGGERED Getting Host spare Adding VMKernel Adding Host to inventory Adding networks Adding filers Done
Howto ?
Detection
my $host = "192.168.2.51"; eval { my $ping = 1; while ($ping) { `/bin/ping -c 1 $host`; if ($? != 0) { $ping = 0; last; } sleep 1; print "Host $host pings\n"; } alarm(0); }; print "Failed to ping\n"; print "Getting Hosts spare \n ";
Howto ?
Adding IP to Host
my $vmkernelIp = "192.168.2.53"; my $vmkernelNetmask = "255.255.254.0"; my $vlanId = "1234";
my $HostSystemView = Vim::find_entity_view( 'view_type' => 'HostSystem' ); my $NetworkSystemView = Vim::get_view( 'mo_ref' => $HostSystemView->configManager->networkSystem );
my $HostPortGroupSpec = HostPortGroupSpec->new( 'name' => "VMKernel", 'vlanId' => $vlanId, 'vswitchName' => "vSwitch0", 'policy' => HostNetworkPolicy->new(), ); $NetworkSystemView->AddPortGroup( 'portgrp' => $HostPortGroupSpec );
Howto ?
Adding IP to Host
my $HostIpConfig = HostIpConfig->new( 'dhcp' => 'false', 'ipAddress' => $vmkernelIp, 'subnetMask' => $vmkernelNetmask ); my $HostVirtualNicSpec = HostVirtualNicSpec->new( 'ip' => $HostIpConfig, ); my $vnic = $NetworkSystemView->AddVirtualNic( 'portgroup' => "VMKernel", 'nic' => $HostVirtualNicSpec );
Howto ?
Adding Host
my $DatacenterView = Vim::find_entity_view( 'view_type' => 'Datacenter', 'filter' => { 'name' => "Rbx" } ); my $ClusterView = Vim::find_entity_view( 'view_type' => 'ClusterComputeResource', 'begin_entity => $DatacenterView, 'filter' => { 'name' => "Cluster001" } ); $hostConnectSpec = HostConnectSpec->new( 'force' => 1, 'hostName' => $vmkernelIp, 'userName' => $hostUsername, 'password' => $hostPassword, 'sslThumbprint' => $thumbprint, ); $ClusterView->AddHost_Task( 'spec' => $hostConnectSpec, 'asConnected' => 1, 'license' => $licenseKey );
Howto ?
Adding storage
my $HostViews = Vim::find_entity_views( 'view_type' => 'HostSystem', 'begin_entity' => $DatacenterView ); my $DatastoreSystemView = Vim::get_view( 'mo_ref' => $HostView->configManager->datastoreSystem, ); my $HostNasVolumeSpec 'accessMode' 'localPath' 'remoteHost' 'remotePath' 'type' ); = HostNasVolumeSpec->new( => 'readWrite', => nas-000357, => '192.168.2.10', => /share/nas-000357', => 'NFS'
Howto ?
Adding vlans
my $HostSystemView = Vim::find_entity_view( 'view_type' => 'HostSystem', 'filter' => {'name' => "192.168.2.53",}, ); my $HostSystemView = Vim::find_entity_view( 'view_type' => 'HostSystem' ); my $NetworkSystemView = Vim::get_view( 'mo_ref' => $HostSystemView->configManager->networkSystem, ); my $HostPortGroupSpec = HostPortGroupSpec->new( 'name' => "vlanXXX", 'vlanId' => 1234, 'vswitchName' => "vSwitch0", 'policy' => HostNetworkPolicy->new(), ); $NetworkSystemView->AddPortGroup( 'portgrp' => $HostPortGroupSpec );
Win
\o/
2 AM
vCenter crash
Alert received
Quick diagnosis
vCenter logs
Panic: Win32 exception: Access Violation (0xc0000005) Read (0) at address 0000000000000098 Panic: Assert Failed: "openedInfo._activations == 0" @ d:/build/ob/bora340981/bora/vim/lib/vimutil/view/multiContainerView.cpp:123 Backtrace: backtrace[00] rip 000000018010a8aa Vmacore::System::Stacktrace::CaptureWork backtrace[01] rip 00000001800e8008 Vmacore::System::SystemFactoryImpl::CreateFileWriter backtrace[02] rip 00000001800e84fe Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace backtrace[03] rip 0000000180129ba5 Vmacore::PanicExit backtrace[04] rip 0000000180129cb1 Vmacore::PanicExit backtrace[05] rip 000000018007669e Vmacore::RunTimeFailure backtrace[06] rip 000000013fddd883 (no symbol) backtrace[07] rip 000000013fdcae13 (no symbol) backtrace[08] rip 0000000000eef1e1 Vim::View::ListView::_GetType backtrace[09] rip 0000000000358d9c Vmomi::RuntimeFault::_GetDataType backtrace[10] rip 000000014044cf6f (no symbol) backtrace[11] rip 000000014042a672 (no symbol) backtrace[12] rip 0000000140430a08 (no symbol) backtrace[13] rip 000000018011f68d Vmacore::System::IsEnlisted backtrace[14] rip 0000000180120a6f Vmacore::System::IsEnlisted backtrace[15] rip 0000000180119df5 Vmacore::System::ThisThreadExists backtrace[16] rip 0000000073562fdf endthreadex backtrace[17] rip 0000000073563080 endthreadex backtrace[18] rip 00000000777bf56d BaseThreadInitThunk backtrace[19] rip 00000000779f3281 RtlUserThreadStart [2012-04-13
02:35:32.385 05324 panic 'App' opID=50E97826-00000154] Assert Failed: "openedInfo._activations == 0" @ d:/build/ob/bora-
340981/bora/vim/lib/vimutil/view/multiContainerView.cpp:123
Quick diagnosis
More verbose ?
C:\Users\All Users\VMware\VMware VirtualCenter\vpxd.cfg <log> <level>trivia</level> </log>
Quick diagnosis
Looks better ... :)
Panic: Win32 exception: Access Violation (0xc0000005) Read (0) at address 0000000000000098 Panic: Assert Failed: "openedInfo._activations == 0" @ d:/build/ob/bora340981/bora/vim/lib/vimutil/view/multiContainerView.cpp:123 Backtrace: backtrace[00] rip 000000018010a8aa Vmacore::System::Stacktrace::CaptureWork backtrace[01] rip 00000001800e8008 Vmacore::System::SystemFactoryImpl::CreateFileWriter backtrace[02] rip 00000001800e84fe Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace backtrace[03] rip 0000000180129ba5 Vmacore::PanicExit backtrace[04] rip 0000000180129cb1 Vmacore::PanicExit backtrace[05] rip 000000018007669e Vmacore::RunTimeFailure backtrace[06] rip 000000013fddd883 (no symbol) backtrace[07] rip 000000013fdcae13 (no symbol) backtrace[08] rip 0000000000eef1e1 Vim::View::ListView::_GetType backtrace[09] rip 0000000000358d9c Vmomi::RuntimeFault::_GetDataType backtrace[10] rip 000000014044cf6f (no symbol) backtrace[11] rip 000000014042a672 (no symbol) backtrace[12] rip 0000000140430a08 (no symbol) backtrace[13] rip 000000018011f68d Vmacore::System::IsEnlisted backtrace[14] rip 0000000180120a6f Vmacore::System::IsEnlisted backtrace[15] rip 0000000180119df5 Vmacore::System::ThisThreadExists backtrace[16] rip 0000000073562fdf endthreadex backtrace[17] rip 0000000073563080 endthreadex backtrace[18] rip 00000000777bf56d BaseThreadInitThunk backtrace[19] rip 00000000779f3281 RtlUserThreadStart [2012-04-13 02:42:32.385 05324 panic 'App' opID=50E97826-00000154] Assert Failed: "openedInfo._activations == 0" @ d:/build/ob/bora-340981/bora/vim/lib/vimutil/view/multiContainerView.cpp:123
Help ?
Google kb.vmware.com => S/R my.vmware.com
Solution
Quick, easy and efficient
vpxd.exe -b
Cool?
Quick diagnosis
CMDB
Storage definitions vCenter defintions Hosts definitions
Lets do it
Lets do it
I got Redis that is a Folder Working on vmAndTemplates we are on a root context
Entity is neither host neither vm neither datastore - _whatToDoWithContextAndPathForThisEntity I have to create a container (Folder) named Redis in vmAndTemplates (root) -> vmAndTemplates
_whatToDoWithContextAndPathForThisEntity Directly creating Folder in vmAndTemplates Finding view of entity Folder, named Redis beginning from vm done Applying alarms No alarms on this entity, skipping Now, looking at what's inside Folder Redis I got, from this at root, 4 direct entity
I have to move a vm named redis100 in Redis (Folder) in path vmAndTemplates/Redis
moving vm into Redis context is Folder Task to move redis100 into Redis (vmAndTemplates/Redis) started Applying alarms No alarms on this entity, skipping VM moved I have to move a vm named redis101 in Redis (Folder) in path vmAndTemplates/Redis moving vm into Redis context is Folder Task to move redis101 into Redis (vmAndTemplates/Redis) started Applying alarms No alarms on this entity, skipping VM moved
I have to move a vm named redis102 in Redis (Folder) in path vmAndTemplates/Redis
moving vm into Redis context is Folder Task to move redis102 into Redis (vmAndTemplates/Redis) started Applying alarms No alarms on this entity, skipping VM moved
I have to move a vm named redis103 in Redis (Folder) in path vmAndTemplates/Redis
moving vm into Redis context is Folder Task to move redis103 into Redis (vmAndTemplates/Redis) started Applying alarms No alarms on this entity, skipping VM moved I got SMTP that is a Folder
Finding view of entity ResourcePool, named prod beginning from Web done Applying alarms No alarms on this entity, skipping Now, looking at what's inside ResourcePool prod I got, from this at root, 2 direct entity Entity is neither host neither vm neither datastore _whatToDoWithContextAndPathForThisEntity
I have to create a container (ResourcePool) named Front in prod (ResourcePool) -> hostsAndClusters/Cluster001/Web/prod - _whatToDoWithContextAndPathForThisEntity
Using Pool prod I have to move a vm named web1057 in Front (ResourcePool) in path hostsAndClusters/Cluster001/Web/prod/Front I have to move a vm named web1058 in Front (ResourcePool) in path hostsAndClusters/Cluster001/Web/prod/Front moving vm into Front context is ResourcePool Task to move web1058 into Front (hostsAndClusters/Cluster001/Web/prod/Front) started Applying alarms No alarms on this entity, skipping VM moved I have to move a vm named web1055 in Front (ResourcePool) in path hostsAndClusters/Cluster001/Web/prod/Front moving vm into Front context is ResourcePool Task to move web1055 into Front (hostsAndClusters/Cluster001/Web/prod/Front) started Applying alarms No alarms on this entity, skipping VM moved I have to move a vm named web1040 in Front (ResourcePool) in path hostsAndClusters/Cluster001/Web/prod/Front moving vm into Front context is ResourcePool Task to move web1040 into Front (hostsAndClusters/Cluster001/Web/prod/Front) started Applying alarms No alarms on this entity, skipping
Snapshot vCenter
Takes alarms set up on entities Takes config of
Pools / Clusters / vApps
Inventory hierarchy
Folder / Clusters / Pools / vApps
All entities
Snapshot vCenter
- alarms: [] config: !!perl/hash:ResourceConfigSpec cpuAllocation: !!perl/hash:ResourceAllocationInfo expandableReservation: 1 limit: -1 reservation: 1125 shares: !!perl/hash:SharesInfo level: !!perl/hash:SharesLevel val: normal shares: 4000 memoryAllocation: !!perl/hash:ResourceAllocationInfo expandableReservation: 1 limit: -1 reservation: 125 shares: !!perl/hash:SharesInfo level: !!perl/hash:SharesLevel val: normal shares: 163840 name: Internal type: ResourcePool childs: - alarms: [] config: vmx: '[nas-000357] web100/web100.vmx' name: web100 type: vm values: [] - alarms: [] config: vmx: '[nas-000357] web101/web101.vmx' name: web101 type: vm values: []
VMware Solution
My own experience
Bring added value
Questions ?
Download this presentation on
www.ovh.com/vmworld